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Coordinating  Information  and  Decisions  of  Hierarchical  Distributed  Decision  Units  in 

Crises 

Introduction:  Balancing  Two  Core  Requirements 

The  survival  of  groups  and  organizations  depends  on  two  key  processes.  First,  they 
must  achieve  extremely  high  levels  of  proficiency  and  efficiency  on  routine  activities 
such  as  the  production  of  products  and  services,  maintenance  of  machines  and 
software,  and  responding  to  feedback  from  clients  or  customers.  This  requires  reliance 
on  shared  goals  and  beliefs,  standardized  patterns  of  behavior,  convergent  thinking, 
and  intense  concentration  on  executing  plans.  Second,  at  the  same  time,  organizations 
must  also  provide  the  capability  for  very  rapid  and  appropriate  responses  to  crises  and 
adversity  (Milburn,  et  al.,  1983a,  1983b)  and  adaptation.  This  requires  early  recognition 
of  potential  threats  and  opportunities,  accurate  and  comprehensive  understanding  of 
new  situations,  divergent  thinking,  and  (at  least  some)  unplanned  behavior.  Effective 
organizations  find  ways  to  balance  the  demands  for  efficiencies  in  routine  activities  and 
readiness  to  notice  and  respond  effectively  to  unexpected  threats.  Both  efficiency  and 
readiness  depend  upon  the  development  of  effective,  high  quality  individual  and 
collective  decision  making  processes  (Herek,  at  al.,  1987). 

Examination  of  organizational  decision  processes  reveals  that  they  vary  widely  but  have 
several  common  characteristics  -  they  tend  to  restrict  innovation,  limit  the  number  of 
ideas  generated  and  possible  alternatives  considered,  and  perpetuate  doubtful 
practices  (Nutt,  1984).  Sometimes  it  is  quite  apparent  that  emergencies,  crises,  and 
disasters  are  consequences  of  poor  decision  making  processes.  The  Cuban  missile 
crisis  (Brugioni,  1991)  and  Challenger  (Feynman,  1992;  Joyce,  1986;  McConnell,  1986; 
Smith,  1986a,  1986b;  U.S.  Presidential  Commission,  1986)  cases  are  forceful 
reminders  of  this  fact.  Similar  motivation  to  better  understand  and  improve  collective 
decision  making  is  provided  by  experiences  like  Xerox’s  failure  to  exploit  its  lead  in 
personal  computers  and  local  area  networks  (Smith,  1988).  Perhaps  too  infrequently 
there  are  equally  forceful  reminders  that  inadequate  decision  processes  not  only  cause, 
but  exacerbate,  existing  crises,  as  in  the  Vincennes  (Appendix  1)  example  (Burns, 

1988;  Moore,  1988;  United  States  Congress  House  Committee  on  Armed  Services 
Subcommittee  on  Investigations,  1993;  United  States  Congress  Senate  Committee  on 
Armed  Services,  1988). 

The  overall  objective  of  this  project  is  to  achieve  an  improved  understanding  of 
organizational  decision  processes  designed  to  achieve  high  levels  of  performance  in 
terms  of  both  routine  and  crisis  challenges.  Better  understanding  requires  both  a 
systemic  perspective  and  strong  inductive  methods.  Thus,  a  combination  of  three 
methods  were  used  in  this  project:  macro-analysis,  simulation  and  experimentation. 
Macro-analysis  is  designed  to  assess  the  quality  and  coverage  of  past  studies  in  order  to 
guide  two  forms  of  simulation  research  aimed  at  discovering  better  designs  for 
organizational  decision  systems.  One  simulation  form  is  a  computer  model  of  the  key 
variables;  the  other  is  a  simulator  with  distributed  people  performing  multiple  decision 
tasks. 
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Strategies  for  Balancing  Efficiency  and  Readiness 

Relationships,  decision  processes,  and  behaviors  useful  for  achieving  efficiency  are 
likely  to  be  dysfunctional  for  readiness.  The  analyses,  attention  to  details,  automation, 
training,  communication  nets  and  message  content,  and  related  activities  which  help 
assure  efficiency  can  hinder  responses  to  adversity  and  crises  for  two  major  reasons. 
First,  achieving  efficiency  consumes  resources,  at  least  some  of  which  could  be 
otherwise  used  to  achieve  readiness  to  cope  with  crises.  Second,  because,  by 
definition,  routine  activities  consume  most  of  the  organizational  members’  time,  provide 
most  of  the  feedback  and  learning  opportunities  they  experience,  and  are  continuously 
emphasized  by  leaders  and  policies  like  TQM,  attention  is  not  focused  on  potential 
threats  but  on  incremental  adjustments  to  current  routines.  Similarly,  achieving  high 
levels  of  readiness  to  respond  to  potential  threats  consumes  resources  for  activities  like 
training  and  intelligence,  resources  which  could  othenwise  be  devoted  to  improving 
efficiency.  Also,  achieving  readiness  focuses  attention  more  on  the  organization’s 
environment  than  on  internal  processes. 

One  way  organizations  attempt  to  address  the  tension  between  internal  (efficiency)  and 
(largely)  external  (readiness)  demands  is  specialization.  To  address  efficiency 
concerns  organizations  rely  on  resources  and  tactics  like  production  engineers,  self- 
directed  work  teams  and  just-in-time  inventory  systems.  To  prepare  for  and  cope  with 
threat  they  rely  on  activities  like  strategic  planning  and  crisis  management  training  (of 
public  relations  and  safety  experts).  The  result  is  poor  integration  of  the  subsystems 
addressing  efficiency  and  readiness,  often  followed  by  the  organization’s  demise  (Miller, 
1990). 

How  can  a  more  appropriate  balance  of  attention  and  other  organizational  resources  be 
achieved?  Is  organizational  learning  the  answer?  TQM?  Trial  and  error?  Training? 
For  many  reasons,  too  numerous  to  fully  elaborate  here,  these  and  similar  popular 
theses  cannot  provide  clear  guidance.  For  example,  learning  opportunities  come  form 
two  primary  sources,  failure  and  imitation.  By  their  nature,  routine  failures  occur  more 
frequently  than  failures  due  to  crises.  Thus,  left  to  everyday  experiences, 
organizational  members  commonly  learn  more  about  how  to  achieve  efficiency  than 
how  to  assure  readiness  for  responding  to  major  adversity  or  crises.  Also,  learning 
from  crises  may  teach  the  wrong  lessons.  This  is  because  members’  responses  to 
chaotic  environments  may  themselves  create  "second  order"  chaos  among  members 
as  they  try  to  coordinate  less  familiar  activities  under  severe  time  pressure  (Ackoff, 
1974).  Learning  is  further  inhibited  because  the  increased  chaos  is  difficult  to  trace 
back  to  or  attribute  to  the  well  intentioned  but  inappropriate  responses  (Diehl  and 
Sterman,  1995).  Even  training  exercises  which  help  members  practice  appropriate 
responses  to  contrived  chaotic  conditions  are,  at  best,  a  partial  remedy.  Training  to 
improve  readiness  for  one  kind  of  crisis  (e.g.,  conventional  war  between  two  countries; 
product  tampering)  may  interfere  with  improving  readiness  for  another  type  of  crisis 
(e.g.,  guerrilla  war  between  a  technologically  advanced  country  and  adversaries  who 
resemble  civilians;  unexpected  death  of  a  key  person).  Further,  those  who  design 
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training  exercises  cannot  be  expected  to  fully  anticipate  the  intensity,  novelties,  or  vital 
subtleties  of  future  chaotic  challenges  (Gettys,  et  al.,  1987). 

This  project  emphasizes  the  decision  process  as  the  critical  issue  for  achieving  better 
balance.  Whereas  organizations  facing  the  challenge  of  finding  highly  efficient  routines 
may  function  best  using  a  centralized  decisions  process,  so  too  may  organizations 
facing  crises.  On  the  other  hand,  in  complex  situations,  whether  routine  or  in  crisis,  a 
decentralized  system  may  work  best.  Where  the  same  organization  or  units  face  both 
the  efficiency  and  readiness  challenges  they  may  be  required  to  quickly  shift  not  only 
their  attention,  but  their  procedures  and  structures.  Development  and  maintenance  of 
multiple  decision  making  systems  coupled  with  a  meta-rule  for  selecting,  and  perhaps 
improving,  the  most  relevant  decision  system,  is  thus  a  serious,  even  fundamental, 
organizational  challenge. 

Nature  of  Empirical  Evidence 

It  would  be  expected  that  social  science  research  can  help  find  the  appropriate  balance. 
Unfortunately,  researchers’  agendas  and  resources  produce  only  oblique,  and  perhaps 
misleading,  implications.  There  are  many  reasons  why  social  scientists’  theories  and 
findings  have  had  limited  relevance.  First,  and  probably  most  important,  their  work, 
particularly  their  experimental  efforts,  generally  fails  to  account  for  the  systemic  nature 
of  organizations.  Bertalanffy’s  (1968)  general  systems  ideas,  and  Miller’s  (1978) 
comprehensive  guide  to  the  nature  of  groups  and  organizations  as  living  systems  have 
not  been  much  used  as  guideposts  by  most  researchers.  For  example  Miller’s 
emphasis  on  (positive  as  well  as  negative)  feedback,  abrupt  discontinuities,  adaptive 
capabilities,  and  complex  interactions^  among  elements  are  only  partially  reflected  in 
the  experiments  of  an  extremely  small  number  of  scholars  (e.g.,  Diehl  and  Sterman, 
1995).  More  typically, 

“[t]he  practice  of  looking  at  isolated  variables  has  contributed  to  a  state  of  affairs 
where  factors  that  make  big  differences  in  [collective]  behavior  are  held  constant 
because  they  are  neither  easily  controlled  nor  manipulated  in  the  laboratory  and 
where  potentially  important  intervening  variables  are  often  ignored.”  (Foushee, 
1984) 

Systemic  analyses  demand  simultaneous  consideration  of  numerous  variables.  As 
Ackoff  (1974)  noted,  to  examine  parts  of  a  system  in  the  hope  of  understanding  the 
whole  is  futile  because  it  is  the  many  and  complex  interactions  among  the  parts  which 
are  critical,  not  just  simple  effects  of  one  variable  on  another.  In  contrast  to  the  large 
number  of  interacting  factors  with  which  organizational  members  must  cope,  the 


^  Discussing  the  central  role  of  interaction  effects  as  reasons  why  good  intentions  cause  crises  to  become 
more  intense,  Perrow  (1984,  p.9)  noted,  "...[T]he  interaction  of  small  failures  led  them  to  construct  quite 
erroneous  worlds  in  their  minds..."  [leading  to  actions  resulting  in  extremely  serious  accidents].  (Emphasis 
added.) 
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majority  of  social  science  investigations,  particularly  experimental  efforts,  consider  only 
the  effects  of  one  or  two  variables  on  a  very  limited  number  of  dependent  measures.^ 

To  rely  on  such  simple  studies  to  obtain  an  accurate  understanding  of  large  systems 
requires  valid  assumptions  about  how  to  aggregate  research  results.  It  is  tempting  to 
assume  simple  effects  combine  in  an  additive  manner.  However,  increasing  evidence 
from  the  physical  sciences  (Gleik,  1987)  demonstrates  that  small  changes  over  some 
ranges  of  even  a  single  variable  can  cause  linear  changes  as  well  as  sudden 
discontinuities  and  chaotic  patterns  in  another  variable.  It  seems  unreasonable  to 
expect  the  same  is  not  true  for  social  systems  as  well. 

A  second  major  factor  limiting  the  ability  of  researchers  to  guide  organizational 
balancing  efforts  is  that  their  evidence  is  more  uneven  than  their  theories.  The  result  is 
an  asymmetry  of  research  evidence  across  the  individual,  group,  and  organizational 
levels  of  systems.  As  will  be  documented  by  the  results  of  this  project,  of  these  three 
levels  of  analysis,  the  ovenwhelming  majority  of  studies,  particularly  experimental 
studies,  deal  with  individuals.'^  However,  in  work  and  other  organizational  contexts, 
collaboration  is  endemic;  isolated  individual  activities  are  necessary  but  not  sufficient 
behaviors.  It  is  true  that  some  theorists  (e.g.,  Perrow,  1984;  Weick,  1989,  1990,  1993) 
emphasize  linkages  within  organizational  systems.  To  date,  relatively  few  experimental 
researchers  have  found  the  resources  or  exhibited  the  patience  to  follow  their  lead.® 

This  is  unfortunate.  Performance  on  collective  tasks  can  rarely  be  predicted  by 
knowledge  of  the  performance  of  individual  members  (Jones,  1974).  In  addition, 
failures  in  complex  situations  like  flight  crews  face  in  actual  (Cooper,  et  al.,  1979)  and 
simulator  (Smith,  1979)  emergencies  are  more  likely  due  to  defective  group  processes 
rather  than  insufficient  (individual)  expertise  or  technical  proficiency  of  crew  members 
(Foushee,  1984;  Foushee  and  Helmreich,  1987).  Yet,  there  were  very  few  studies  of 
group  behavior  for  over  15  years  beginning  in  about  1960  (Zander,  1979). 


^  This  is  probably  because  psychologists  have  dominated  most  of  the  relevant  domains.  And 
“Psychologists  ...  tend  to  look  only  for  solutions  to  problems  that  lend  themselves  to  [their]  methodological 
approaches.”  (Coovert,  etal.,  1995). 

^  There  are  many  other  variables  and  experiments  which  could  be  added  to  this  argument.  For 
example,  the  nature  and  complexity  of  a  team  member’s  information  processing  is  affected  by  the 
nature  of  the  immediate  task  (its  size,  time  pressure,  etc.  -  Payne,  Bettman  and  Johnson,  1988), 
the  member's  integrative  complexity  (Streufert,  et  al.),  the  member's  judgment  policy  (Brehmer 
and  Brehmer,  1988),  not  to  mention  order  of  arrival  of  specific  data,  the  member's  experience  or 
expertise,  type  of  response  required  (estimate  or  choice),  availability  of  decision  aids,  and 
numerous  other  well  documented  factors. 

*  This  is  particularly  true  of  research  on  assessing  and  dealing  with  risk  (Clarke  and  Short,  1993). 

Studies  of  intra-group  communication,  inter-group  conflict,  coalition  formation,  and  free  ridership  in 
social  contexts  are  obvious  exceptions  to  the  focus  on  individuals.  However,  no  researchers 
attempt  to  conduct  experiments  at  the  organizational  level. 

®  Concerned  with  numerous  instances  of  commercial  airplane  crashes,  two  leading  researchers  surveyed 
studies  of  crew  training  and  performance.  They  concluded,  “[Pjractically  all  pilot  ...performance  research 
has  focused  on  individual ...  skills.”  (Kanki  and  Foushee,  1989). 
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Even  if  there  were  as  many  experiments  conducted  at,  for  example,  the  organizational 
level  as  there  are  now  at  the  individual  level,  it  could  be  dangerous  to  rely  on  findings  to 
help  organizations  balance  the  demands  of  routine  and  readiness.  The  reason  is  that 
factors  at  the  different  levels  of  living  systems  (Miller,  1978)  may  have  significant  joint 
effects  (Tomer,  1992).  Because  studies  rarely  incorporate  variables  from  more  than 
one  level,  these  effects  are  largely  unexamined.® 

A  third,  related,  limitation  of  existing  research  evidence  is  that  the  most  systemic 
treatments  rely  on  the  weakest  induction  techniques.  Anthropological  and  case 
analyses  (e.g.,  Perrow,  1984;  Rochlin,  1989;  Roberts,  1990;  McKinney,  1993; 
Eisenhardt,  1989;  Miller,  1990;  Feynman,  1992),  and  survey  research  do  provide 
insights  about  possible  causes  of  failures  and  successes  in  complex  decision  making 
settings.  For  example,  Perrow  (1984)  used  very  detailed  analyses  of  documents  and 
interviews  to  document  his  argument  that  organizational  members’  shared  experience 
and  perspectives,  while  minimizing  the  need  for  communications  to  coordinate 
dispersed  activities,  can  hinder  the  causal  interpretation  of  complex  events  and 
subsequent  problem  solving  during  crises.  Weick  (1993)  relied  on  existing  documents 
to  support  his  thesis  that,  in  the  confusion  of  chaotic  conditions,  members  find  it  difficult 
to  sense  subtle  and  unforeseen  causal  links  between  changes  in  the  states  and 
processes  of  a  system’s  element.  However,  the  causal  propositions  offered  by  these 
kinds  of  studies  need  to  be  experimentally  verified  if  they  are  to  provide  solid  guidance 
for  administrators  (Doherty,  1993).^ 

The  asymmetry  in  experimental  research  is  likely  to  continue.  Experimenters  exploring 
the  dynamics  of  organizational  and  group  processes  face  extraordinary  challenges. 
Each  observation  requires  several  person-hours.  Experimental  tasks  must  reflect  key 
attributes  of  organizational  settings.  When  simulators  are  used,  as  in  the  third  phase  of 
this  project,  vast  amounts  of  data  are  generated,  requiring  substantial  data  reduction 
(for  example,  coding  of  communications)  systems.  Researchers’  personal  incentives 
(e.g.,  tenure  and  raises)  are  little  concerned  with  rigorous  experimental  testing  of  macro 
propositions. 

A  fourth  reason  to  rely  on  existing  evidence  very  cautiously  is  that  social  scientists, 
particularly  those  who  use  experimentation,  prefer  static  to  dynamic  contexts,  especially 
when  considering  organizational  level  issues.  For  example,  organizational  change  is 
typically  considered  in  terms  of  evolutionary  rather  than  radical  shifts  in  practices  and 
properties  (Barnett,  1995).  Yet  for  organizations,  in  both  routine  and  crisis  modes, 
change  is  the  norm.  Organizational  changes  are  erratic,  often  unplanned,  locally  as 
much  as  systemically  inspired,  and  sometimes  dysfunctional.  Even  the  few 


®  As  an  example,  most  researchers  who  consider  group  decision  making  do  not  incorporate  either 
organizational  contexts  and  variables  (e.g.,  the  nature  of  the  organization’s  technology)  or  individuals’ 
characteristics  (e.g.,  mental  ability)  in  their  designs. 

^  Often  this  will  necessitate  the  use  of  quasi-experimental  studies  (Campbell  and  Stanley,  1963). 


ARI  -  MDA-903-90-C-0154  -  Final  Report  Page  6 


experimenters  who  approach  dynamics  of  these  kinds  do  so  with  extremely 
oversimplified  designs  and  challenges  for  their  subjects.® 

It  could  be  argued  that  these  objections  to  the  utility  of  past  research  ignore 
experiments  involving  groups  and  teams.  It  is  true  that  there  has  long  been  interest  in 
what  makes  groups  or  teams  effective.  Some  of  the  earliest  research  on  groups 
focused  on  how  different  leadership  arrangements  affected  group  performance  (Lewin, 
at  al.,  1939).  A  continuing  concern  has  been  the  problem  solving  and  decision  making 
performance  of  groups  (e.g.,  Shaw,  1932;  Shaw,  1954;  Steiner,  1972;  Davis,  1992; 
Hollenbeck  at  al.,  1995).  Several  group  decision  making  studies  emphasized  the 
effects  of  work  loads,  stress,  or  time  pressure  on  decision  processes  and/or 
performance  (e.g.,  Stouffer,  et  al.,  1949;  Isenberg,  1981;  Wright;  1984;  Calloway, 
Marriott,  and  Esser,  1985;  Stasser  and  Titus,  1987;  Ediand,  1994;  Kerstholt,  1994; 
Urban  et  al.,  1995;  Caccaro,  Gualtieri,  and  Minionis,  1995;  Neck  and  Moorehead, 

1995). 

Despite  this  history  of  group  studies,  “we  still  know  little  about  why  some  groups 
perform  better  than  others  (Foushee,  1984).”  In  addition  to  the  factors  that  limit  the 
relevance  of  past  research  to  the  problem  of  balancing  efficiency  and  readiness,  there 
are  unique  problems  in  group  research.  One  is  that  even  social  psychologists  very 
often  ask  individual  subjects  to  react  to  “paper  people”  or  videotaped  situations  rather 
than  investigating  the  interactions  between  two  or  more  people.  In  studies  where 
participants  actually  interact  in  experimental  contexts,  subjects’  tasks  lack  the 
complexities  encountered  in  attempting  to  find  an  organizational  balance  between 
efficiency  and  readiness.  Usually  there  is  but  one  task  to  perform,  only  one  phase  of 
the  decision  process  is  examined,  the  task  changes  little  if  at  all  over  trials,  or  there  is 
no  opportunity  to  alter  the  interpersonal  and  organizational  features  of  the  decision 
making  system.  The  number  of  variables  considered  is  small.®  Because  each 
experiment  uses  a  somewhat  different  subset  of  variables,  inconsistent  results  are  not 
uncommon.  Even  if  these  problems  were  not  present,  experiments  on  groups  may 
have  limited  relevance  for  achieving  better  balances  of  routine  and  readiness  in 
organizations.  It  has  been  estimated  that  95  percent  of  all  studies  of  groups  have  been 
conducted  outside  “natural”  settings,  mostly  in  laboratories  (Guzzo  and  Shea,  1990). 
Consequently  there  are  numerous  pertinent  issues  which  remain  to  be  addressed 
(Tuler,  1988). 

The  preceding  arguments  are  well  known  but  not  taken  seriously.  While  experimental 
social  scientists  often  acknowledge  in  their  discussions  that  organizations  are  large 
systems  consisting  of  subsystems  that  are  meshed  in  many  ways  (e.g.,  task 
coordination,  information  flows,  interpersonal  affective  bonds),  and  while  they  further 

®  Researchers  like  Diehl  and  Sterman  (1995)  in  MIT’s  system  dynamics  groups  appear  to  have  made  the 
most  progress  in  this  respect.  Even  their  designs  involve  minimal  complexity  compared  to  actual 
organizational  settings. 

®  As  argued  earlier,  this  tends  to  preclude  understanding  of  systemic  effects.  In  this  context,  one 
researcher  concerned  with  aircrew  performance  noted,  “it  is  difficult  to  assimilate  the  sheer  number  of 
variables  than  can  potentially  affect  group  processes."  (Foushee,  1984) 
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recognize  that  people  (individually  and  collectively)  are  subsystems  capable  of 
intelligent  self  design,  these  perspectives  continue  to  too  rarely  inform  their  empirical 
research.  If  the  resources  consumed  by  research  are  to  be  fully  exploited,  to  speed 
scientific  progress  and  increase  its  practical  applications,  more  holistic  projects  are 
necessary.  Fortunately,  with  advances  in  technology  (e.g.,  computer  hardware  like 
parallel  processing),  software  (e.g.,  neural  networks),  and  statistical  procedures  (e.g., 
meta-analysis),  the  feasibility  of  more  comprehensive  research  and  assessments  of 
bodies  of  research  is  growing.  This  project  sought  to  exploit  and  accelerate  these 
trends. 

Systemic  Research  Methods 

This  project  used  three  approaches  in  an  attempt  to  overcome  many  of  the  problems  in 
the  literature.  The  first  was  macro-analysis.  It  is  a  new,  systematic  method  for 
assessing  how  thoroughly  all  relevant  aspects  of  complex  issues  have  been  examined 
by  researchers.  The  second  approach  was  to  develop  a  language  capable  of 
simulating  organizational  decision  processes,  both  routine  and  in  crises.  The  third  was 
to  enhance  and  exploit  an  organizational  simulator  to  conduct  experiments.  Each  of 
these  approaches  will  be  explained  by  way  of  introducing  each  of  the  three  major 
phases  of  this  project.  The  importance  of  using  these  more  systemic  methodologies 
becomes  more  emphatic  when  the  properties  of  organizational  crises  are  taken 
seriously.  Therefore  the  methodologies  will  be  addressed  following  reflections  on  the 
nature  of  organizational  crises. 

Organizational  Crises 

Emergencies,  disasters,  accidents  and  crises  are  immensely  important  events  in 
business  (Fink,  1986;  Meyers,  1986;  and  Miller,  1984)  and  military  (Smith  and  Asker, 
1993;  Roberts,  1990)  The  flightpath  ...,  1988)  and  other  (Eisenhardt,  1993;  Foushee, 
1984)  organizations  because  they  can  negate  all  the  benefits  of  long  histories  of 
successful  routine  efforts. 

As  noted  earlier,  decision  making  in  most  organizations  consists  of  relatively  long 
periods  of  routine  activities  interrupted  by  infrequent  crises.  In  contrast  to  routine 
decisions,  crises  typically  involve  surprise,  extremely  early  deadlines,  threat  -  the  real 
possibility  that  the  most  important  goals  (including  survival)  will  not  be  fully  understood 
or  satisfied,  high  levels  of  ambiguity,  one  or  more  key  decision  makers  unavailable, 
(potentially)  extreme  individual  and  interpersonal  stress  (Herman,  1963;  Herman,  et  al., 
1974;  Staw  et  al.,  1981)  and,  some  (e.g..  Turner,  1976)  would  argue,  very  ill-structured 
challenges. 

Crises  arise  from,  and  are  accelerated  by,  many  sources  (Milburn  et  al.,  1983a),  several 
of  which  may  affect  an  organization  simultaneously.  Clearly  those  organizations  whose 
members  lack  the  imagination  or  time  to  take  the  “impossible”  seriously,  or  who  have 
become  complacent,  or  have  become  highly  focused  (e.g.,  on  achieving  extremely  high 
efficiencies  in  routine  operations,  or  on  those  activities  which  generated  earlier 


ARI  -  MDA-903-90-C-0154  -  Final  Report  Page  8 


successes  -  Miller,  1990)  are  subject  to  crises.  However,  there  should  be  equal 
concern  with  less  emphasized  sources. 

Crises  are  often  inherent  in  the  design  of  complex  systems  which  are  capable  of 
producing  effects  not  anticipated  by  their  designers  and  incompletely  understood  by 
their  operators  (Perrow,  1984).  Crises  can  also  arise  because  decision  makers’ 
reactions  to  events  are  founded  on  misconceptions  about  causes  and  effects. 
Misconceptions  may  be  more  numerous  and  more  severe  where  relevant  data  or 
decision  makers  are  “distributed”  (at  different  locations),  and/or  where  they  lack 
appropriate  decision  aids  (Sage,  1987).  The  very  (“dominant”  -  Staw,  et  al.,1981) 
beliefs  and  behaviors  learned  during  routine  activities,  as  well  as  those  acquired  in 
attempting  to  cope  with  earlier  crises,  can  cause  or  exacerbate  crises.  It  has  been 
observed  that,  “[Ujnlearning  may  be  sometimes  a  more  difficult  short-term  task  than 
learning”  (Smart  and  Vertinsky,  1977).  Insufficient  variety  can  cause  crises  if 
appropriate  responses  to  complex  circumstances  require  correspondingly  complex 
organizational  capabilities  which  have  not  been  developed  or  maintained  (Ashby, 

1956).  Contagion  may  also  be  a  factor.  Messages  containing  even  a  hint  of  worry  or 
stress,  particularly  if  they  are  numerous  and  arrive  at  accelerating  rates  from  many 
sources,  may  produce  a  vigorous  sensation  of  panic  or  crisis.  (Levy  and  Nail,  1993; 
Hatfield,  et  al.,  1994) 

Examples  of  all  these  sources  of  crises  can  be  found  in  human  creations  like 
technology  (e.g.,  Boeing’s  “fly  by  wire”  technology  in  the  777),  economic  activity  (e.g., 
unauthorized  trades  in  futures  markets),  and  geopolitical  events  (e.g.,  the  Suez  crisis  of 
1962).  In  addition,  it  is  important  to  recognize  a  fourth  human  source  of  crises.  Their 
number,  frequency  and  intensity  depends  on  the  quality  of  both  routine  and  crisis 
decision  making  activities.  Poor  routine  decisions  foster  crises.  Poor  decision  making 
during  crises  exacerbate  rather  than  resolve  crises. 

Both  routine  and  crisis  decisions  are  handicapped  by  lack  of  imagination^®  and 
misunderstanding.  In  his  analyses  of  “normal  accidents”  Perrow  (1984)  offered  Three 
Mile  Island’s  nuclear  power  plant  as  an  example.  Those  who  designed  and  constructed 
the  plant  were  not  the  same  people  who  operated  and  controlled  it.  Despite  extensive 
training  and  rehearsals  in  simulated  emergencies,  operators’  grasp  of  what  could 
happen  in  many  contingencies  was  limited  or  non-existent.  Perrow’s  minute  by  minute 
documentation  of  reactions  to  the  leak  at  Three  Mile  Island  revealed  that  engineers’ 
beliefs  about  what  was  happening  in  the  system  were  sometimes  the  opposite  of  actual 
events.  Hence,  at  times,  their  discussions  and  decisions  not  only  failed  to  help  control 
the  plant,  but  increased  its  instability. 

Even  where  technologies  are  better  understood,  crises  may  arise  from  the  cumulative 
effects  of  small,  virtually  undetectable,  events  (Gersick  and  Hackman,  1990;  Weick, 
1990).  For  example,  Weick  (1990)  argues  that,  among  other  apparently  minor  factors. 


A  spokesperson  for  Japan's  Mihamam  nuclear  plant  said,  “We  never  thought  that  the  radiation  could 
increase  so  quickly  because  such  a  thing  hadn’t  happened  before.” 
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small  changes  in  plans  or  routines  coupled  with  regression  to  first  (and  best)  learned 
behaviors  under  stressful  conditions,  “hearing”  what  one  is  anticipating  will  be  said 
(rather  than  seeking  clarification  of  confusing  messages),  and  reluctance  of 
inexperienced  participants  to  question  superiors  were  keys  to  the  crash  (on  the 
ground!)  of  two  747’s  and  the  loss  of  nearly  600  lives. 

Perrow  and  Weick  agree  that  serious  crises  are  most  likely  in  “tightly  coupled”  systems. 
Tightly  coupled  systems  have  very  many  elements,^ ^  few  if  any  redundancies,  and 
complex  but  strong  cause-effects  linkages.  “Coupled”  is  misleading  because  it 
seriously  understates  amplification.  As  Weick  emphasizes,  where  success  requires  the 
small  events  to  occur  in  proper  sequence,  failure  of  one  event  to  take  place  at  the  right 
time  and  as  planned  greatly  multiplies  the  risks  of  severe  failure  of  the  larger  system. 
Weick  also  notes  that  loosely  coupled  systems  may  very  suddenly  become  extremely 
tightly  coupled  when  many  “small”  events  have  unrecognized  but  amplified 
consequences. 

Well  intentioned  decisions  and  actions,  particularly  when  taken  in  the  context  of 
incompletely  understood  systems,  may  cause  new  crises  or  increase  the  severity  of 
existing  crises.  For  example,  when  the  Chernobyl  nuclear  plant  was  being  operated  at 
reduced  power  during  a  routine  maintenance  operation,  experiments  were  undertaken. 
Although  it  remains  unclear  exactly  what  the  purpose  of  the  experiments  was,  they 
involved  investigation  of  the  effects  of  changes  in  control  rod  positions  on  the  safety 
system’s  performance.  A  United  Press  (1986)  report  noted: 

"The  investigators  think  the  operator  pulled  out  some  control  rods  and  upset  (the) 
delicate  balance....  'Alarmed,  he  moved  some  more  rods  to  try  to  get  the  situation 
back  under  control  -  and  this  caused  part  of  the  reactor  to  'go  critical.’"' 

Even  when  people  are  well  intentioned,  when  they  seek  to  avoid  crises  and  minimize 
their  adverse  effects,  they  create  organizational  and  institutional  arrangements  which 
initiate  or  expand  crises  (Clarke  and  Short,  1993).  A  vivid  example  was  the  Chicago 
flood  of  1992  which  resulted  in  more  than  $1  billion  in  damages  and  lost  income  (Bailey 
and  Burton,  1992;  Freak  flood  ...,  1992).  Early  in  the  century,  tunnels  beneath  the  city’s 
center  were  dug  in  order  to  provide  deliveries  of  goods  and  removal  of  waste  without 
adding  to  the  congestion  of  surface  transportation  systems.  In  1991 ,  when  a  “rational” 
activity,  the  replacement  of  decayed  wooden  bumper  posts  which  protected  bridges 
over  the  Chicago  River  from  being  struck  by  ships  and  barges,  was  not  carefully 
planned  and  monitored,  the  replacement  piles  were  driven  through  the  bed  of  the 
Chicago  River  and  top  of  one  of  the  delivery  tunnels.  It  should  be  noted  that 
complacency,  arrogance  or  incompetence  also  helped  cause  this  crisis;  authorities  had 
been  warned  (with  photos  and  video  tape)  about  leakage  from  the  river  into  the  tunnels. 
Remedial  action  had  been  delayed  because  the  city  sought  additional  bids  which  were 


”  Perrow  (1984)  emphasizes  six  sources  of  failure  in  tightly  coupled  systems  with  the  acronym  DEPOSE, 
Design,  Environment,  Procedures,  Operators,  Sources/materials,  and  Equipment.  Of  these,  at  least  60 
percent  of  most  crises  or  accidents  in  complex  or  advanced  technology  systems  are  attributed  to  human 
components,  the  operators. 
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lower  than  those  originally  submitted.  "It's  a  lot  of  little  actions  that  led  up  to  a  disaster," 
said  John  Laplante  (the  later  fired)  Director  of  Transportation.  "A  lot  of  what  happened 
are  honest  and  competent  employees  making  errors  of  judgment." 

Crises  often  are  the  result  of  cumulative  ineffectiveness  which  is  difficult  to  detect. 

There  may  be  no  single  or  "large"  weakness  in  the  decision  system.  Those  "small" 
weaknesses^^  which  are  present  are  not  fatal  in  isolation,  and  they  are  often 
manageable.  Many  decision  system  features  are  double  edged  but  viewed  solely  as 
assets.  For  example,  the  development  of  shared  mental  models,  while  reducing 
communications  needs,  also  reduce  hypothesis  generation  capability,  reducing 
creativity  and  innovation.  Similarly,  interaction  (Stasser,  et  al.,  1989;  Heath  and 
Gonzalez,  1995),  or  shared  or  redundant  (Slovic,  1982)  data  can  help  cut  costs  and 
time  for  search  but  tend  to  unconsciously  inflate  confidence.  In  short,  despite  the 
presence  of  positive  attributes  in  organizational  systems,  there  is  a  real,  perhaps  large, 
possibility  that  "small"  weaknesses  aggregate  in  complex  (e.g.,  multiplicative)  ways, 
cumulating  in  a  crisis. 

An  example  of  the  multiplicative  impact  of  small  events  in  tightly  coupled  human  and 
technological  systems,  and  of  good  intentions  gone  awry,  involved  the  electrical  power, 
telephone,  and  airline  industries.  Late  in  the  afternoon  of  September  17,  1991,  AT&T 
suffered  a  loss  of  power^^  which  forced  the  air  traffic  controllers  in  the  New  York  city 
area  to  shut  down  most  of  their  operations. Controllers  in  New  York  lost  90  percent 
of  their  communications  capability  with  adjacent  on  Long  Island.  Thus,  New  York 
controllers  could  not  "hand  off"  flights  that  were  leaving  the  200  mile  radius  for  which 
they  were  immediately  responsible.  They  also  lost  50  air  to  ground  frequencies  used  to 
communicate  directly  with  pilots.  As  a  direct  result  1,174  flights  carrying  85,000 
passengers  were  canceled  or  delayed;  100  planes  sat  idle  on  runways  for  four  hours.^® 
Later  the  FAA  claimed  there  was  no  safety  threat  to  passengers  and  crews. 

How  did  this  crisis  arise?  Explanations  focused  on  four  causes. 

1)  An  earlier  FAA  request  to  the  General  Service  Administration  (GSA)  for  parallel, 

redundant  systems  using  AT&T  and  (e.g.,  MCI)  had  been  denied  by  GSA  on  the 
basis  of  cost. 

2)  To  try  to  save  money,  AT&T  had  an  "interruptable  power"  contract  with  Con  Ed. 

3)  The  afternoon  of  September  17th  was  unexpectedly,  unseasonably  hot  causing  huge 
air  conditioning  loads  on  Con  Ed’s  generating  and  distribution  systems.  Con  Ed 
decided  to  interrupt  power  for  firms  with  contracts  like  AT&T’s. 

4)  AT&T  responded  to  Con  Ed's  notice  that  power  would  be  cut  by  switching  to  its  own 
generators. 


"[Gjreat  events  have  small  beginnings."  (Perrow,  page  9.) 

This  failure  was  despite  several  billion  dollars  spent  by  AT&T  to  assure  a  "self-healing"  system. 

This  was  the  third  time  since  January  1990  in  which  a  major  failure  in  the  AT&T  system  was 
experienced  (at  least  one  of  the  earlier  ones  was  due  to  the  use  of  new  software  which  contained  a  bug). 

Airport  and  airline  operations  were  only  part  of  the  story.  470,000  international,  and  4.5  million 
domestic,  calls  didn't  get  through.  Some  of  them  certainly  were  emergency  calls. 
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The  generators  were  not  on  the  same  floor  as  the  one  occupied  by  those  monitoring 
the  system.  So  when  rectifiers  (which  change  AC  to  CD  current)  blew,  switching 
AT&T’s  system  to  battery  operation,  monitors  were  relying  on  a  warning  system.  At 
the  time,  that  warning  system  was: 

a)  Not  operating  properly  (some  audio  warnings  were  muffled,  some  lights  were  not 
functioning  or  had  been  disabled),  and 

b)  Ignored  by  AT&T’s  operating  personnel  and  their  supervisor  because  they  were 
at  a  training  session  learning  about  a  new  warning  system,  and 

c)  Ignored  by  managers  who,  substituting  for  the  trainees,  failed  to  assure  that 
previously  specified  procedures  were  followed.. 

After  six  hours  of  use  AT&T’s  emergency  batteries  had  not  been  replenished,  power 
fell,  and  telephone  switching  equipment  failed. 


Fortunately,  the  concatenation  of  a)  “good”  decisions  (to  economize  -  AT&T  and  the 
GSA,  and  to  provide  technical  training  -  AT&T)  with  b)  extreme  conditions,  and  c)  tight 
couplings  among  the  three  industries  did  not  cause  a  major  airline  disaster. 

Decisions  made  during  crises  may  also  intensify  the  crises  due  to  reliance  on  mindless 
use  of  information  (Langer,  1992),  an  inability  to  shift  from  automatic  to  conscious 
cognitive  activities  (Louis  and  Sutton,  1991),  habitual  thinking  and  processes  (Gersick 
and  Hackman,  1990),  and  inflexibility  or  organizational  routines  (Weiss  and  llgen, 

1985).  Even  substantial  departures  from  normal  conditions  may  not  be  sufficient  to 
alter  rigid  patterns  of  thinking  and  interaction.  It  has  been  argued  that  the  crash  of 
Florida  International’s  flight  90  into  the  14th  Street  Bridge  in  Washington,  DC  (National 
Transportation  Safety  Board,  1982)  was  such  a  case.  Of  course,  crises  typically 
increase  time  pressures  on  decision  makers,  and  would  be  expected  to  cause  a  shift  to 
less  rigid,  more  mindful  (Langer,  1989)  processes.  However,  in  at  least  one  study  in  a 
dynamic  environment,  decision  makers  relied  on  the  same  decision  making  strategy  no 
matter  how  much  time  pressure  they  experienced  (Kerstholt,  1994). 

Social  and  economic  systems  contain  not  only  complex  technologies,  but  webs  of 
causes  and  effects  which  are  often  less  salient  than  technological  linkages.  Thus,  in 
the  realm  of  economic  and  business  decisions  it  is  reasonable  to  expect  that  the 
combination  of  complex  systems  and  inappropriate  understanding  of  those  systems  will 
frequently  exacerbate  crises.  It  has  long  been  documented  that  instability  in  complex 
situations  is  caused  by  decisions  founded  on  misinterpretations  of  feedback  data.  For 
example,  in  the  classic  beer  game  experiments,  participants’  attribute  cyclical  demand 
to  fluctuations  in  consumers’  decisions,  whereas  demand  changes  only  once  (Sterman, 
1989).  Even  though  participants  are  very  intelligent,  are  subject  to  market  discipline, 
and  have  strong  performance  incentives,  and  even  though  the  system  they  are  asked 
to  manage  is  quite  simple,  their  choices,  not  the  single  exogenous  shock  the  system 
receives  early  in  the  experiment,  create  and  sustain  instability. 

In  the  areas  of  diplomacy  and  international  conflict  it  is,  unfortunately,  common  to  find 
instances  where  crises  originated  or  were  escalated  by  decision  makers  who  were  well 
intentioned  but  unable  to  adequately  grasp  the  social  and  geopolitical  elements  of  the 
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situation.  McNamara’s  observations  about  the  decisions  made  in  the  Kennedy  and 
Johnson  administrations  reveal  that  the  sacrifices  made  during  the  Vietnam  war  were 
magnified  by  top  decision  makers’  ignorance,  misconceptions,  and  misinterpretations 
(McNamara,  1995).  Social  reinforcements  for  these  cognitive  problems  were  provided 
by  syncopates  and  peers,  compounding  the  problems  and  crises. 

Experienced  decision  makers  know  that  crises  arise  from  subtle  combinations  of 
nonsalient  and  unanticipated  causes,  and  that  effective  responses  depend  on  early 
recognition,  and  appropriate  responses.  They  know  teams  and  organizations  can  fall  in 
to  “self-fueling  spirals”  (Hackman,  1990).  Their  performance  tends  to  decline  (improve) 
if  they  somehow  begin  on  a  good  poor  (good)  track.  For  example,  in  fighting  forest  fires 
early  detection  coupled  with  very  quick  and  very  well  learned,  appropriate  responses 
are  the  keys  to  controlling  damage  and  minimizing  the  resources  used  to  control  the 
crisis  (Klein,  1976;  Klein,  and  Calderwood,  1991;  Mclean,  1992; ).  Appropriate 
responses  demand  expert  knowledge  about  how  different  combinations  of  fuels,  terrain, 
temperature,  humidity,  and  other  “technological”  considerations  affect  the  crisis. 
Moreover,  “social”  knowledge  is  critical.  Among  other  things,  appropriate  decisions  by 
a  commander  require  assessments  of  human  endurance;  the  current  locations  and 
capabilities  of  team  members;  their  knowledge  of  local  terrain;  and  how  effectively 
“headquarters”  will  be  able  to  marshal  additional  supporting  resources  (such  as  weather 
forecasts  for  the  immediate  area,  relief  personnel,  air  tankers,  and  satellite  photos). 

Correct  interpretations  of  events  and  of  the  utility  of  decision  alternatives  is  likely  to  be 
hindered  when  superiors  fail  to  seek  and  use  their  subordinates’  ideas  and  information. 
Among  the  more  subtle  factors  that  may  influence  the  willingness  to  use  subordinates’ 
inputs,  particularly  those  which  disagree,  is  the  superior’s  assessment  of  the  situation 
(Weick,  1993).  One  study  found  that  when  the  situation  was  viewed  by  a  superior  as 
either  a  crisis  or  as  a  minor  issue,  subordinates’  contradictory  inputs  were  unlikely  to  be 
used  to  make  constructive  use  of  their  suggestions  (Tjosvold,  1984).  However,  when 
superiors  regarded  the  situation  as  a  challenge,  subordinates’  information  and 
dissenting  opinions  tended  to  improve  the  superior’s  decisions. 

Crises  may  also  be  caused  or  escalate  because  decision  makers  do  not  initially  analyze 
matters  using  complex  integrations  of  perspectives,  or  because  their  integrative 
complexity  declines  as  perceived  threat  increases,  or  because  they  feel  little  or 
declining  accountability  to  others  (Levi  and  Tetlock,  1980;  Mandel,  et  al.,  1993;  Walker 
and  Watson,  1994). 

Contrast  this  abbreviated  litany  of  the  nature  and  sources  of  crises  (or  review  Appendix 
1)  with  the  earlier  characterization  of  research.  It  is  hoped  that  the  comparison  has 
motivated  two  things:  1)  the  reader’s  interest  in  research  methods  which  provide  better 
integration  of  the  many  relevant  variables,  and  more  rigorous  inductive  treatment  of 
complex  systems. 

Macro  Analysis  of  Relevant  Research  -  Phase  1 
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Macro-  and  Meta-Analysis 

The  first  phase  of  the  project  was  labeled  a  “macro-analysis”  to  distinguish  its  objectives 
from  those  of  meta-analyses.  The  concern  is  epistemological.  Macro-analysis 
procedures  were  developed  as  a  complement  to  meta-analysis  and  to  encourage  more 
symmetry  and  thoroughness  in  research  on  systems  like  distributed  decision  making 
systems.  The  purpose  of  meta-analysis  is  to  estimate  the  true  size  of  effects  on  a 
dependent  variable  (e.g.,  judgmental  accuracy)  caused  by  variations  in  one  or  more 
independent  variables  (Hunter,  and  Schmidt,  1990,  1991,  1994).  Effect  sizes  are 
adjusted  for  factors  in  the  relevant  set  of  studies  which  can  bias  or  affect  the  magnitude 
of  effects  in  any  one  study  (e.g.,  restriction  of  range  and  sample  size).  In  essence, 
each  study  is  one  observation  in  the  meta-study  of  all  the  studies  which  investigate  the 
same  hypothesis.  Meta-analyses  have  proved  valuable  in  resolving  apparent  conflicts 
in  studies’  findings,  achieving  not  only  more  accurate  estimates  of  effect  sizes,  but  also 
identifying  situations  in  which  moderator  variables  may  be  affecting  effect  sizes. 

Meta-analyses  cannot  be  conducted  on  any  issue  where  there  are  only  a  few  studies  of 
the  issue.  Thus,  the  set  of  meta-analyses  found  in  a  literature  is  an  implicit  indicator  of 
the  research  focus  of  social  scientists.  However,  meta-analysis  does  not  systematically 
identify  research  issues  which  could  or  should  be  subjected  to  strong  inductive 
treatment.  To  systematically  identify  important  unresearched  issues  it  is  necessary  to 
conduct  “macro-analyses”  of  past  research.^®  Macro-analyses  are  particularly 
important  where  scientists  are  attempting  to  understand  complex  systems  which  require 
analysis  by  multidisciplinary  teams. 

Macro  Analysis  Procedures 

In  many  important  ways  the  steps  of  macro-analysis  are  quite  different  from  those  in 
meta-analyses.  Meta-analysis  has  a  reductionist  quality  in  that  it  usually  examines 
pairs  of  variables.^^  In  contrast,  macro-analysis  emphasizes  systems  of  relationships 
among  pairs  of  variables  as  well  as  interactions  between  two  or  more  variables. 
Whereas  both  kinds  of  analysis  require  discovery  of  relevant  research,  relevance  is 
defined  very  differently.  Macro-analysts  must  determine  which  variables  are  to  be 
included.  In  contrast,  in  the  most  basic  meta-analysis  the  two  variables  in  a  proposition 
or  hypothesis  of  interest,  as  well  as  their  synonyms,  are  used  as  key  words  when 
searching  the  titles,  abstracts,  keyword  lists,  and  texts  of  research  reports.  As  will  be 
seen,  many  more  variables  are  involved  in  macro-analyses.  Normally  meta-analyses 
scan  not  only  studies  in  peer  reviewed  journals,  but  technical  reports  as  well.  Macro¬ 
analyses  must  focus  on  research  published  in  peer  reviewed  journals  to  remain 
feasible.  Once  a  body  of  relevant  studies  is  identified,  meta-analysis  codes  each 
study’s  characteristics  (e.g.,  sample  size)  and  transforms  data  where  appropriate  (e.g. 


There  are  other  advantages  of  macro-analysis  as  well.  It  can  document  the  extent  to  which  specific 
methodological  treatments  and  paradigms  extend  across  disciplines  and  time. 

This  oversimplifies  somewhat.  Meta-analyses  concerned  with  moderator  variables  or  interaction  effects 
involve  more  than  two  variables. 
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from  correlation  coefficient  to  difference  score),  before  conducting  more  statistical 
analyses  to  adjust  raw  effect  sizes  into  estimates  of  actual  effect  sizes. 

Macro-analysis  parallels  early  steps  in  meta-analysis  but  stops  before  relevant  studies 
are  coded.  Macro-analysis  begins  by  identifying  a  system,  parts  of  which  have  been 
the  subject  of  (experimental)  research  but  which  is  still  imperfectly  understood.  The 
next  step  involves  developing  a  list  of  variables  which  reasonable  people  believe  are 
pertinent  to  the  set  of  issues  or  system  of  interest.  The  list  of  variables  should  be  as 
exhaustive  as  possible.  To  encourage  the  development  of  a  comprehensive  list, 
macro-analysis  must  review  more  than  experimental  or  other  research.  It  is  important 
to  read  cases,  journalistic  and  technical  reports,  historical  narratives,  and  documents  in 
files.  It  is  also  important  to  question  participants  who  have  been  involved  in  relevant 
experience.^®  As  the  list  of  variables  develops  care  must  be  taken  to  identify  synonyms 
and  antonyms  to  avoid  redundancy.  In  many  systems  involving  human  actors  two  or 
more  variables  interact  to  affect  a  dependent  variable.  In  such  cases  the  interacting 
variables  may  be  treated  as  a  single  (joint)  variable  in  the  key  list. 

The  next  step  in  macro-analysis  is  to  construct  a  square  matrix  using  the  list  of  key 
variables  twice,  once  for  the  rows  and  again  for  the  columns.  Each  row  will  contain  all 
references  to  studies  in  which  a  variable  has  been  treated  as  an  independent  variable 
in  a  study.  Each  column  will  contain  all  references  to  studies  in  which  a  variable  has 
been  a  dependent  variable.  Thus  each  cell  represents  a  simple  research  proposition 
involving  a  pair  of  key  variables.  When  an  experiment  containing  at  least  one  pair  of 
independent-dependent  variables  in  the  list  of  key  variables  has  been  identified,  the 
experiment’s  index  number  is  entered  in  the  appropriate  cell(s)  of  the  matrix.  Thus,  an 
experiment’s  index  number  would  be  entered  in  two  cells  if  that  study  examined  the 
effects  of  status  incongruency  on  satisfaction  with  a  team’s  decision  process  and  also 
examined  the  effects  of  satisfaction  with  decision  process  on  status  congruency. 
Ultimately  each  cell  of  the  matrix  will  contain  an  unique  number  for  each  study  which 
examines  the  cell’s  hypothesis. 

After  the  numerous  variables  related  to  the  focal  system  have  been  identified,  it  is 
necessary  to  identify  relevant  studies.  Prior  to  the  introduction  of  computerized  data 
bases  this  was  not  feasible  for  issues  involving  large  systems  or  many  variables. 
However,  with  the  development  of  computerized  abstracts  in  PsycLIT^®  it  is  possible  to 
conduct  reasonably  efficient  searchers  using  Boolean  logical  operations. 

Once  all  studies  are  indexed  and  their  index  numbers  have  been  entered  in  appropriate 
cells,  several  insights  can  be  obtained  simply  by  visual  inspection.  One  is  the  density  of 
researchers’  attention  to  variables  and  hypotheses.  It  is  immediately  apparent  that 
where  there  is  a  high  density  cell,  that  is,  where  there  are  numerous  studies  of  an 


Many  good  examples  of  this  kind  of  canvassing  of  participants  can  be  found  in  Mowen,  1993. 
This  data  base  now  contains  abstracts  for  journal  articles  from  1967  to  the  present.  It  also 
references  book  chapters  from  1967  to  1987  and  abstracts  book  chapters  published  since  1987. 
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hypothesis,  a  meta-analysis  is  likely  to  be  fruitful.^®  A  row  with  few  index  numbers 
entered  in  any  of  the  cells  may  indicate  an  independent  variable  that  either  needs  more 
study,  is  implicitly  judged  to  be  of  limited  relevance  by  researchers,  or  is  simply  very 
difficult  to  examine. 

Overlays  of  multiple  matrices  can  be  particularly  useful.^^  Suppose  that  one  wished  to 
address  the  idea  that  research  in  laboratories  with  ad-hoc  student  groups  provided 
different  conclusions  than  research  relying  on  “real”  people  in  on-going  teams  in  field 
settings.  If  an  overlay  of  the  two  matrices,  one  for  each  type  of  study,  revealed  different 
patterns  of  density  for  one  type  than  for  the  other,  it  would  indicate  that  conclusions 
from  one  setting  cannot  be  generalized  to  the  other.  Another  useful  overlay  is  to  have 
three  matrices,  one  for  studies  finding  no  effects,  another  reporting  positive  effects,  and 
a  third  reporting  negative  effects  regarding  a  specific  cell’s  research  question. 

Overlaid  matrices  can  examine  method  variance  effects  by  comparing  the  loci  of 
studies  in  one  matrix  summarizing  experiments  with  another  summarizing  survey 
research.  Or  a  researcher  concerned  with  differences  between  the  effects  of  noise  in 
communications  on  small  group  and  organizational  judgmental  accuracies  might  use 
one  matrix  for  small  group  studies  and  a  second  matrix  for  organizational  studies. 
Multiple  matrices  can  also  be  useful  when  paths  of  effects  among  dependent  variables 
are  considered. 

Transformations  of  the  original  matrix  may  also  prove  useful.  Where  definitions, 
theories,  or  clear  empirical  patterns  can  justify  doing  so,  certain  variables  can  be 
aggregated  to  provide  better  insights  about  what  can  be  concluded  from  a  literature. 

An  example  of  this  approach  was  used  in  this  project  and  will  be  discussed  shortly. 

Macro-Analysis  of  Phase  1 

The  macro-analysis  in  the  first  phase  of  this  project  had  two  main  objectives.  The  first 
was  to  prioritize  future  research  by  identifying  gaps,  redundancies  and  conflicts  in 
previous  studies.  The  second  was  to  discover  standards,  variables,  and  relationships 
to  be  used  in  construction  of  simulations  in  the  second  phase  of  the  project. 

More  than  80  key  words  or  phrases^^  (in  Appendix  3)  related  to  distributed  decision 
making  in  routine  and  crisis  settings  were  identified.  This  was  accomplished  by  reading 
a  wide  variety  of  sources  in  addition  to  the  experimental  literature  on  decision  making.^ 


Even  where  there  has  been  a  meta-analysis  of  a  cell’s  hypothesis,  the  density  may  suggest  a  newer 
one  could  be  conducted  to  incorporate  studies  not  included  in  the  most  recent  meta-analysis. 

With  overlaid  matrices  it  may  prove  useful  to  develop  mathematical  comparisons,  such  as  simiiarity 
measures.  This  project  did  not  require  such  sophisticated  capabilities. 

There  were  originaliy  69  key  words  or  phrases.  New  ones  were  added  as  the  literature  search 
progressed  to  assure  a  complete  canvass  of  the  literature. 

Unfortunately,  as  is  obvious,  not  aii  these  sources  were  available  at  the  time  the  list  of  key  variabies 
and  phrases  was  being  developed.  This  section  includes  only  examples  of  the  many  sources  used  during 
the  project.  For  a  complete  listing  consult  the  references  section  of  this  report. 
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Among  the  major  sources  were  detailed  accounts  of  events  like  Wal-Mart’s  growth 
(Trimble,  1990;  Vance  and  Scott,  1994),  the  Cuban  missile  crisis  (Janis,  1972,  1982, 
and  1989;  Janis  and  Mann,  1977;  Anderson,  1983;  Brugioni,  1991;  Sylvan  and 
Thorson,  1992;  Guttieri  et  al.,  1995),  NASA’s  decision  to  launch  Challenger  (Covault, 
1986;  Joyce,  1986;  Smith  1986a  &  1986b;  US  Presidential  Commission,  1986; 
McConnell,  1987;  Browning,  1988;  Kolcum,  1988;  Feynman,  1992);  Xerox’s  decision 
not  to  market  it’s  “office  of  the  future”  (Smith,  1988),  the  Chicago  flood  (Freak 
flooding...,  1992;  Bailey  and  Burton,  1992),  forest  fires  (Mclean,  1992;  Weick,  1993; 
Sahagun,  1994),  the  Navy’s  shooting  down  of  a  civilian  airliner  (Burns,  1988;  Moore, 
1988;  The  flightpath  to  disaster,  1988;  US  Congress  Senate  Committee...,  1988,  and 
House  Committee...,  1993;  Rogers,  1992),  Chernobyl  (Cohen,  1987;  Brandsjo,  1988; 
Martinez-Val,  1990),  Three  Mile  Island  (Perrow,  1984)  and  other  high  technology 
emergencies  (Tuler,  1988),  aircrews  in  crises  (National  Transportation  Safety  Board, 
1982;  Foushee,  1984;  Helmreich  et  al.,  1985;  Foushee  and  Helmreich,  1987;  Lanir, 
1989;  Ginnett,  1990,  1993;  Weick,  1990;  Deitz  and  Thoms,  1991;  Helmreich  and 
Foushee,  1993;  McKinney,  1993;  Lavin,  1994),  infrastructure  failures  (e.g.,  Lavitt,  1991; 
McKenna,  1991),  and  operation  of  high  reliability  systems  (Cooper  et  al.,  1979;  Rochlin, 
1989;  Weick,  1989;  Roberts,  1990;  Clarke  and  Short,  1993;  Eisenhardt,  1993;  Weick 
and  Roberts,  1993).  Ethnographic  and  case  studies  (e.g.,  Nutt,  1984;  Mulder  at  al., 
1986;  Hickson,  1987;  Kuklan,  1988;  Prechel,  1994)  were  examined.  In  addition, 
literature  reviews  and  meta-analyses  (e.g.,  Maass  and  Clark,  1984;  Brehmer  and 
Brehmer,  1988;  Levy  and  Nail,  1993;  Lopes,  1994;  Mullen  et  al.,  1994;  Salas  et  al., 
1995;  Guzzo  and  Dickson,  1996),  simulation  articles  (e.g.,  Carley,  1986;  Brannick,  et 
al.,  1993;  Coovert,  1995),  editorial  opinions,  theories  and  models  (e.g.,  Hertzler,  1940; 
Mackenzie,  1976;  Billings  etal.,  1980;  Heiner,  1988;  Kaplan,  1983;  Miao,  1991;  O’Hare, 
1992;  Klein,  1993;  Lewis  and  Sycara,  1993;  Proulx,  1993;  Barnett,  1995),  and  essays 
(e.g..  Churchman,  1971;  Milburn,  1977;  Smart  and  Vertinsky,  1977;  Staw  et  al.,  1981; 
Milburn  et  al.,  1983;  Miller,  1984;  Fischhoff,  1985;  Langer,  1989,  1992;  Eisenberg, 

1990;  Gersick  and  Hackman,  1990;  Beach  and  Lipshitz,  1993;  Larson,  1993;  Thietart 
and  Forgues,  1995)  have  been  unsystematically  monitored  since  1988. 

In  a  series  of  Boolean  searches  of  all  studies  from  1983^^^  to  1991  in  PsycLIT,^®  60,888 
unique,^®  potentially  significant,  studies  were  identified  and  downloaded.  Each  abstract 
was  then  screened  to  determine  whether  or  not  the  study  should  be  retained  for  further 


When  this  project  began  Psychological  Abstracts  did  not  include  abstracts  for  studies  prior  to  1983. 

The  macro-analysis  was  supplemented  by  on-line  searches  of  PsycLIT  as  pre-1983  and  post-1991 
abstracts  were  added  to  the  PsycLIT  data  base.  The  additional  studies  from  those  periods  did  not  change 
the  results  of  the  original  macro-analysis. 

Many  other  sources  were  consulted,  but  they  proved  less  complete  or  redundant  with  PsycLIT.  Among 
those  sources  were  the  following  on-line  data  banks:  ABI/INFORM  (management  and  administration), 
ERIC  (education,  measurement,  personnel  training),  NTIS  (social  sciences),  SOCIAL  SCIRESEARCH 
(social-  behavioral  sciences),  and  SSIE  (social  and  engineering  sciences). 

No  single  search  could  cope  with  the  entire  set  of  key  words.  The  searches  excluded  all  studies  not 
involving  adults.  Over  250  searches  were  ultimately  conducted  to  assure  a  thorough  examination  of  all 
relevant  research.  Duplication  was  eliminated  using  specially  written  computer  programs.  Other  custom 
programs  were  used  to  streamline  abstracts  for  faster  and  more  accurate  reviews. 
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analysis.  Twenty  one^’^  criteria  (Appendix  4)  were  used  to  delete  irrelevant  studies.  A 
sample  of  abstracts  was  evaluated  by  at  least  two  trained  coders  and  audited  by  the 
author  to  assure  consistent  treatment.  Custom  programs  were  used  to  remove 
redundant  or  irrelevant  studies  whenever  possible.  These  activities  resulted  in  1 ,523 
studies  to  be  reviewed.^®  Simultaneously  articles  in  many  scientific  journals  (Appendix 
5)  were  studied.  Those  articles  deemed  to  be  relevant  were  compared  to  the  articles 
identified  by  the  PsycLIT  search  process  to  assure  the  search  was  thorough.  No 
oversights  were  discovered. 

The  number  of  studies,  on  average,  in  each  cell  of  the  first  macro-analysis  matrix 
constructed  in  Phase  1  was  fewer  than  0.5,  suggesting  that  most  issues  relevant  to 
distributed  decision  systems  functioning  in  crises  contexts  had  received  minimal 
attention.  However,  none  of  the  logical  or  theoretical  relationships  between  the  80 
variables  in  the  matrix  had  yet  been  used  to  sharpen  the  focus.  Two  consolidations 
were  performed.  First  studies  based  on  synonyms  were  placed  in  the  same  category. 
Second,  the  variables  were  divided  in  to  two  sets,  those  emphasizing  aspects  of  the 
decision  process,  and  those  that  focused  on  the  constraints  and  context  within  which 
the  process  takes  place,  or  on  the  consequences  of  the  processes.  (These  two  sets 
are  listed  in  Appendix  6.)  A  second  matrix  was  constructed  using  the  first  set  as 
columns  and  the  second  as  rows. 

While  working  with  the  second  matrix  it  became  obvious  it  would  not  be  possible  to 
develop  a  meaningful  propositional  inventory  for  two  reasons.  The  first  was  that  the 
modal  cell  in  the  first  matrix  was  empty.  That  is,  there  had  been  no  study  of  most 
possible  pairwise  relationships  among  key  variables.  Where  there  was  more  than  one 
study,  there  were  very  few  and  their  results  were  usually  inconsistent.  These  limitations 
will  be  evident  in  other  sections  of  this  report.  For  example  in  the  discussion  of  stress, 
a  series  of  propositions  is  provided.  Readers  will  note  that  most  of  those  propositions 
are  founded  on  a  single  study  and  that  coverage  of  issues  is  very  uneven. 

The  second  reason  a  meaningful  inventory  was  not  possible  is  that  abstracts  in  PsycLIT 
contained  too  little  information  to  index  most  studies.  Thus,  to  complete  a  propositional 
inventory  would  have  required  each  member  of  the  research  team  to  read  and  code 
more  than  500  studies  in  addition  to  their  other  project  activities.  In  an  attempt  to 
achieve  as  much  closure  as  possible  a  third  matrix  was  developed.  It  consolidated  the 
52  rows  of  the  second  matrix  in  to  eight  categories  (which  are  defined  at  the  bottom  of 
Appendix  6).  The  following  table  indicates  the  number  of  studies  which  could  be  clearly 
coded  in  to  one  of  the  cells  of  the  third  matrix.^® 


Two  were  added  to  the  original  19.  Studies  must  have  been  experimental  and  published  in  peer 
reviewed  journals. 

In  the  following  discussion  additional  studies  are  included  due  to  searches  that  became  possible  in  later 
years  of  the  project. 

®  Note  that,  unlike  the  first  macro-analysis  matrix  which  addressed  a  single  proposition  in  each  cell,  a  cell 
in  this  table  may  contain  studies  of  many  different  hypotheses.  Thus,  even  those  cells  with  several 
studies  usually  do  not  imply  the  possibility  of  a  useful  meta-analysis. 
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Table  1 

Categorization  of  Relevant  Studies 


Accuracy 

Risk 

Environ¬ 

ment 

Mental 

Model 

Crisis 

Outcomes 

Collective 

TOTALS 

Assessment 

21 

11 

5 

10 

7 

3 

44 

101 

Choice 

84 

161 

56 

86 

32 

100 

261 

780 

Communicate 

19 

8 

7 

11 

16 

0 

74 

135 

Estimation 

16 

42 

1 

8 

8 

15 

32 

122 

Forecast  / 
Prediction 

29 

18 

2 

30 

12 

3 

58 

152 

Judgment 

39 

133 

0 

10 

5 

1 

43 

231 

Knowledge  / 
Information 

35 

21 

5 

17 

13 

4 

58 

153 

Search 

7 

8 

0 

6 

1 

5 

10 

37 

Memory  / 

Learning 

8 

8 

6 

10 

8 

10 

32 

82 

TOTALS: 

258 

410 

82 

188 

102 

141 

612 

1793 

As  these  data  suggest,  the  macro-analyses  revealed  that  experimental  foundations  for 
recommendations  to  improve  organizational  decision  making  are  seriously  limited. 

There  are  extraordinary  asymmetries.  Inspection  of  the  margins  in  Table  1  immediately 
reveals  that  studies  of  decision  making  in  crisis  contexts  are  unusual,  constituting  fewer 
than  six  percent  of  all  studies  on  aspects  of  decision  making.  Another  striking 
observation  is  that  only  about  one  third  involved  dyads,  groups,  teams  or  some  other 
form  of  collective  activity:  studies  of  individual  cognitive  behavior  received 
disproportionate  effort.  An  equally  disturbing  limitation  is  the  fact  that,  even  though  an 
essential  step  in  effective  team  decision  making  in  crises  is  accurate  definition  of  the 
situation,  fewer  than  two  percent  of  the  experiments  addressed  one  or  more  elements 
(e.g.,  search,  creativity,  attention,  coordination)  of  the  definition  process. 

The  asymmetries  are  even  more  striking  when  the  data  in  Table  1  are  considered  in 
more  detail.  The  cell  representing  studies  of  intra-team  or  intra-organizational 
communications  in  crises  contains  fewer  than  one  percent  of  all  the  studies.  The  same 
is  true  for  the  cell  pertinent  to  search  in  crisis  conditions.  In  contrast,  20  percent  of  all 
the  studies  have  examined  facets  of  judgment,  estimation,  choice  and  forecasting  in 
risky  situations. 

Some  other  asymmetric  results  of  the  macro-analysis  are  not  apparent  in  Tablel 
Studies  with  provocative  findings  (e.g.,  the  interest  team  members  show  in  others'  ideas 
is  greatest  in  moderately  challenging  settings  and  is  lower  in  crises)^®  are  almost  never 


30 

Similar  patterns  were  found  for  expression  of  disagreement  with  positions  opposed  to  one's  own 
position.  These  kinds  of  findings  may  imply  that  team  members  in  crises  can  quickly  find  an 
inappropriate,  but  apparentiy  consensuai,  definition  of  their  situation.  This  was  apparently  the  case  of 
the  technicians  at  Japan's  Mihamam  nuclear  plant  February  9th  and  10th,  1991.  According  to  the 


ARI  -  MDA-903-90-C-0154  -  Final  Report  Page  19 


replicated.  The  modal  experiment  involved  fewer  than  three  independent  variables, 
each  having  just  two  conditions.  Single  level  designs  dominated  even  though  individual 
characteristics  (such  as  amount  of  stress),  team  characteristics  (such  as 
cohesiveness),  organizational  characteristics  (such  as  allocation  of  decentralized  task 
responsibilities)  and  environmental  features  (e.g.,  amount  of  threat)  are  likely  to 
interactively  affect  decisions  processes  in  crises,  there  were  virtually  no  experiments 
using  one  or  more  independent  variables  from  each  of  these  three  levels.  Typically, 
subjects  in  experiments  were  not  required  to  face  the  consequences  of  their  own 
previous  judgments/decisions  in  subsequent  decision  processes  as  is  normal  in  “real 
world  situations.  Most  subjects  were  inexperienced  high  school  or  college  students. 

These  characterizations  of  relevant  studies  appeared  to  be  too  extreme.  Therefore,  to 
assure  that  the  conclusions  from  the  macro-analysis  were  not  overstated  or  misstated, 
a  series  of  additional  searches  of  the  PsycLIT  abstracts  was  conducted.  A  brief 
description  of  one  of  the  additional  searches  will  demonstrate  the  procedure  and  the 
nature  of  the  findings.  It  supposes  a  researcher  is  concerned  with  how  any  set  of 
decision  makers,  whether  or  not  they  are  distributed,  copes  with  the  onset  of  a  crisis. 
Key  activities  at  that  point  include  sensing  the  situation  accurately,  creating  potentially 
useful  action  plans,  and  collaborating  effectively. 

The  following  lexicographic  Boolean  search  was  used  to  identify  studies  relevant  to 
these  concerns.  The  first  two  steps  assure  that  only  studies  of  collective  (e.g.,  dyadic 
or  larger  groups’  or  teams’,  and  organizational)  behaviors  in  the  face  of  severe  time 
pressure,  threat,  or  crisis  were  considered.  The  third  step  assured  that,  of  the  studies 
selected  in  the  first  two  steps,  only  studies  concerned  with  attention,  creativity,  and 
collaboration  issues  were  be  selected. 

Step  1 :  Select  any  study  involving  at  least  one  of  the  following  key  words  or 
phrases. 

Stress,  Anxiety,  Tension,  Threat,  Crisis,  (work)  Load,  Time  Pressure 
(This  step  produced  183  potentially  useful  studies.) 

Step  2:  From  the  studies  identified  in  Step  1  select  any  study  involving  at  least  one 
of  the  following  key  words  or  phrases.^^ 

Team,  Group,  Distributed,  Organization,  Integrate 

(This  step  eliminated  all  but  78  of  the  183  studies  identified  in  step  1.) 

Step  3:  Of  the  studies  remaining  after  Step  2,  select  any  study  involving  at  least 
one  of  the  following  key  words  or  phrases. 

Set  1:  Search,  Attention,  Scan,  Definition,  Formulat*,  Problem  Space 
Set  2:  Creativity,  Innovation,  Hypothesis  Generation,  Invention 
Set  3:  Coordinat*,  Communicat* 


Associate  Press,  they  interpreted  a  warning  signal  as  a  defective  signal  rather  than  an  indication  of 
the  (actual)  radioactive  leak. 

In  fact,  the  “OR”  logical  operator  was  applied  to  all  the  keywords  or  phrases  within  a  given  step  or  set. 
The  “AND”  operator  was  then  used  to  condense  the  three  sets  into  the  final  set  of  potentially  relevant 
studies. 

A  key  word  containing  *  is  used  to  identify  all  studies  with  the  same  stem.  For  example,  Integrati*  will 
identify  all  studies  with  integration,  integrating . 


ARI  -  MDA-903-90-C-0154  -  Final  Report  Page  20 


Set  4:  Cognitive  Complexity,  Cognitive  Style,  Cognitive  Map,  Mental  Model, 
Policy,  Script,  Schema,  Image  Theory,  Experience,  Expertise 
The  third  step  eliminated  all  but  41  of  the  78  studies  which  remained  after  the  second 
step. 

Reading  the  41  studies  revealed  that  fewer  than  half,  19,  were  actually  pertinent  to 
some  aspect  of  the  early  phases  of  collective  coping  with  crises. The  19  studies 
reported  12  dependent  variables^'*  that  were  significantly  affected  by  one  or  more 
independent  variables.  Most  hypotheses  were  addressed  by  only  one  study.  For 
example,  of  the  19  pertinent  studies,  only  one  compared  experienced  with  ad-hoc 
groups  (Kanki  and  Foushee,  1989).  Whether  in  routine  or  crisis  situations,  flight  crews 
that  had  flown  together  had  more  task  oriented  communications,  more  frequently 
sought  to  validate  understanding  of  messages,  and  exhibited  more  participation  by 
subordinate  officers  when  working  in  a  flight  simulator  than  newly  formed  crews. 

Attempts  at  replication  were  the  exception  in  the  19  studies.  Of  the  12  dependent 
variables,  only  two  were  examined  in  more  than  one  study.  In  both  cases  there  were 
only  two  experiments.  In  one  case  the  first  study’s  results  failed  to  be  replicated  by  the 
second.  Hansford  and  Diehl  (1988)  found  no  effect  of  sex  on  idea  generation  whereas 
(Mabry,  1985)  found  males  made  more  suggestions  in  structured  tasks  whereas 
females  made  more  in  unstructured  tasks.  In  the  second  case,  two  studies  by  the 
same  researcher  were  concerned  with  overall  group  performance  in  rule  induction 
tasks,  but  examined  somewhat  different  sets  of  independent  variables  (Laughlin,  1988; 
Laughlin  and  McGlynn  1986).  In  both  experiments  exchanges  of  evidence  fostered 
performance  more  than  exchanges  of  hypotheses.  This  replication  of  findings  was  a 
notable  exception  to  the  general  patterns  found  in  the  macro-analyses. 

Similar  stepwise  Boolean  searches  of  abstracts  were  conducted  for  additional  topics 
(e.g.,  studies  of  which  factors  affect  groups’,  teams’  or  organizations’  effectiveness 
when  making  routine  decisions).  In  every  case  the  number  of  pertinent  studies  which 
were  located  were  very  few.  In  most  cases  each  study  had  an  unique  combination  of 
variables.  Where  two  or  more  (a  very  rare  event)  studies  considered  the  same 
hypothesis,  different  results  were  usually  obtained. 

Due  to  these,  and  other  less  serious,  limitations,  Foushee’s  (1984)  lament  about  our 
ignorance  of  group  behavior  remains  accurate.  It  applies  even  more  to  organizational 
decision  making  in  crises.  Consequently  the  second  phase  of  the  project,  development 
of  ICARUS,  rested  on  the  research  team’s  judgments.  This  was  particularly  true  when 
considering  how  teams  or  organizations  sense  when  there  is  a  crisis,  what  the  nature  of 
the  crisis  is,  and  what  responses  might  be  appropriate. 

A  Boolean  search  for  studies  which  focus  on  how  collective  choices  are  made  in  crises  resulted  in  34 
studies,  only  5  of  which  were  pertinent.  Similar  severe  “shrinkage”  was  experienced  in  all  searches  of  this 
kind. 

The  12  dependent  variables  were:  idea  generation,  overall  communication,  task-related  discussion, 
validation  of  information,  non-task  discussion,  interest  in  opposing  arguments,  disagreement  with 
opposition  positions,  decision  effectiveness,  decision  speed,  group  performance,  and  individual  power. 
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Simulating  Decision  Making  Systems 

In  its  second  and  third  phases  this  research  program  used  two  simulation  approaches 
to  balance  the  competing  demands  for  systemic,  yet  causal  empirical  analyses, 
logistical  challenges,  data  capturing  and  reduction,  and  realism.^®  One  approach, 
which  was  used  in  the  third  phase  and  will  be  described  later,  hereafter  referred  to  as  a 
simulator  (or  ALGO),  was  in  the  tradition  of  experiential  games  and  war  games.  Much 
like  a  flight  simulator,  it  permits  researchers  to  observe  the  decision  making  processes 
and  outcomes  of  participants  who  play  specific  roles  in  an  organizational  context.  The 
second  simulation  approach  was  to  construct  a  simulation  language,  ICARUS,  to  create 
computer  models  of  organizational  decision  making  systems. 

ICARUS  sought  to  incorporate  variables  and  relationships  whose  critical  nature  have 
been  documented  by  prior  empirical  studies  or  rigorous  analyses.  There  were  two 
ultimate  goals.  One  was  to  conduct  “thought  experiments"  (Davis  and  Kerr,  1986)  to 
determine  under  what  conditions  different  simulation’s  results  paralleled  previously 
documented  behavior.  The  second  was  to  guide  the  design  of  experiments  on  the 
organizational  simulator  (ALCO)  which  followed  in  the  last  phase  of  the  project. 

Simulation  techniques  are  widely  used  to  better  understand  how  to  manage  crises 
associated  with  complex  technologies  like  the  space  shuttle  and  nuclear  power  plants 
(e.g.,  Kolcum,  1988;  Martinson  and  Hobbins,  1991).  However,  most  previous  research 
efforts  of  this  kind,  (e.g.,  Carley  and  Prietula,  1994)  have  not  explicitly  relied  on 
empirical  (experimental  or  survey)  research.  Nor  have  they  been  validated  with 
subsequent  empirical  investigations. 

The  advantages  (and  limitations  -  see  Burton  and  Obel,  1984)  of  computer  simulations 
for  studying  the  behavior  and  design  of  organizations  have  long  been  recognized  (Cyert 
and  March,  1963;  Abelson,  1968;  Oron  et  al.,  1984;  Cohen  and  Cyert,  1965).  The  most 
important  advantage  is  integration.  The  number  of  variables  and  relationships 
incorporated  in  a  simulation  can  more  closely  approximate  those  found  in  complex 
systems  like  organizational  decision  making  systems.  Unfortunately,  existing 
simulations  have  made  relatively  small  advances  in  this  sense.  For  example, 
simulations  of  individual  decision  making  processes  do  not  yet  include  other  relevant 
individual  factors  like  emotion  (Carley  and  Prietula,  1994).  Nor  do  models  of  individual 
cognitive  processes  like  SOAR  (Newell,  et  al.,  1989)  contain  organizational  variables  or 
relationships.  Likewise,  there  is  no  notion  of  individual  cognitive  activity  in  the 
organizationally  oriented  Garbage  Can  simulation  (Masuch  and  LaPotin,  1989). 


Analytic  models  (e.g.,  Boettcher  and  Tenney,  1986)  can  add  human  information  processing 
characteristics  to  normative  (in  their  study,  team  theoretic)  models  to  produce  provocative 
hypotheses  about  how  to  structure  distributed  decision  making  processes.  However, 
computational  constraints  require  analytic  models  to  exclude  all  but  a  very  small  number  of 
variables  and  relationships  which  are  present  in  actual  situations.  Thus,  simulations  may  be  the 
only  way  to  encompass  the  complexity  of  real  crisis  management  systems  with  many  team 
members  at  critical  nodes  in  the  network. 
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Simulations  like  Stasser's  (1988)  DISCUSS  model  of  intra-group  information  flows  do 
not  incorporate  inter-group  (intra-team)  information  flows,  team  structure,  or  any  other 
"macro"  variables.  Carley’s  ELM  emphasizes  organizational  features  (distribution  of 
relevant  information  and  alternative  structures)  but  intentionally  minimizes  attention  to 
individual  behavior.  Even  simulation  research  which  explicitly  addresses  individual  and 
organizational  factors  exhibits  minimal  integration  (Carley  and  Prietula,  1994).  A  truly 
comprehensive  simulation  which  encompasses  the  many  variables,  relationships  and 
levels  of  analysis  inherent  in  decentralized  decision  making  teams  is  likely  to  therefore 
provide  surprises  when  used  as  an  experimental  vehicle  (Stasser,  1988). 

A  second  advantage  of  comprehensive  simulations  is  that  their  construction  reveals 
how  vague  and  incomplete  existing  theories  really  are.  These  revelations,  like  the 
surprises,  stimulate  new  insights  and  more  thorough  consideration  of  the  foundations 
upon  which  empirical  studies  are  constructed  and  evaluated.  When  gaps  in  theories  or 
empirical  understanding  are  identified,  construction  of  simulations  require  explicit 
assumptions  which  may  be  used  as  hypothesis  in  experimental  work.  Strategic  choices 
among  alternative  research  designs  and  methods  are  therefore  better  informed. 

A  third  reason  to  use  simulation  is  that  it  can  better  capture  the  critical  dynamics  of 
reality  than  other  research  methods.  Unlike  most  empirical  research,  a  good  simulation 
should  be  capable  of  suggesting  the  consequences  of  cybernetic,  non-linear, 
discontinuous,  and  chaotic  relationships  among  variables  and  processes  which  are 
central  to  organizational  decision  making  activities  from  search  through  choice. 
Research  failing  to  explicitly  deal  with  these  intra-  and  extra-organizational  dynamics 
seems  likely  to  be  irrelevant  at  best,  if  not  actually  dysfunctional,  to  better 
understanding  of  how  real  decision  systems  work  and  how  they  can  be  improved. 

A  fourth  major  advantage  of  the  comprehensive  simulation  is  that,  once  constructed,  it 
can  be  used  for  a  very  wide  variety  of  inexpensive  and  quick  thought  experiments  to 
assess  propositions  and  complex  models  obtained  from  researchers,  on-line  decision 
makers,  and  those  who  carefully  observe  decision  making  processes.  No  other  method 
permits  experimental  treatment  of  such  complex,  dynamic  issues  as  inexpensively  or 
quickly. 

There  are  several  major  disadvantages  of  simulation  as  empirical  research  vehicles. 

By  far  the  most  important  is  the  magnitude  of  the  task.  In  this  project  characteristics  of 
individual  and  collective  behavior  must  be  carefully  specified  and  linked.  And  those 
elements  must  be  linked  to  equally  carefully  defined  environmental  and  task 
characteristics.  Examination  of  existing  simulations^®  reveals  that  they  are  limited  to 


Simulations  range  from  those  stressing  individual  information  processing,  judgment  and 
decision  behaviors  (Hastie,  1988;  Payne,  et  al.,  1988),  through  various  forms  of  collective 
behaviors  (Huesman  and  Levinger;  Hastie  and  Pennington,  1989;  Markovsky,  1987;  Stasser, 
1988,  Hollenbeck,  etal.,  1995),  to  organizational  (Cohen,  1982;  Cohen,  March  and  Olsen,  1972; 
Marsuch  and  LaPotin,  1989)  decision  making  systems.  The  project  most  like  this  project  appears 
to  be  ACTS,  (Carley  and  Prieula,  1994).  They  are  usually  founded  on,  and  replicate,  behaviors 
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about  six  key  parameters;  too  few  to  capture  the  complexity  of  most  important 
organizational  decisions,  particularly  those  made  during  crises.  Despite  such 
challenges,  in  view  of  the  advantages  of  simulation,  there  is  almost  no  alternative  to  a 
project  like  ICARUS  for  researchers  who  take  the  systemic  nature  of  organizational 
decision  making  seriously  when  constructing  and  testing  theories  which  could  inform 
practice. 

Phase  2  -  ICARUS 

A  goal  in  both  simulation  phases  of  this  project  was  to  avoid  the  necessity  for  “starting 
over”  each  time  a  new  situation  was  to  be  considered.  Therefor  in  the  last  phase  a 
simulator  was  employed  which  could  create  very  large  numbers  of  situations  simply  by 
using  menus  in  a  computer  program;  no  programming  would  be  required  of 
experimenters.  Likewise,  in  the  second  phase  we  sought  to  avoid  construction  of  a 
new  computer  program  for  each  simulated  situation.  So  development  of  simulation 
language,  ICARUS,  was  undertaken.  As  will  be  clear,  this  phase  of  this  project  was 
very  ambitious. 

ICARUS  is  a  language  for  studying  the  response  of  groups  of  intelligent  people  to  crisis 
situations  and  the  evolution  of  their  organization  over  time.  The  ICARUS  language 
allows  the  people,  decision  rules,  and  constraints  of  a  crisis  situation  to  be  described, 
and  the  description  is  used  to  produce  a  trace  of  the  unfolding  of  events  as  the  people 
interact  with  each  other  and  their  simulated  environment.  ICARUS  is  a  tentative  first 
step  in  using  computers  to  model  such  human  interactions.  We  fully  expected  the  heat 
of  the  sun  to  melt  the  wax  holding  the  entire  structure  together.^^ 

General  Framework  Of  Icarus 

The  Ada  programming  language  was  selected  as  the  foundation  for  ICARUS.  Among  its 
many  advantages,  Ada  is  suitable  for  large  scale  simulations,  highly  portable,  and  is 
supported  by  DARPA  -  STARS's  reuse  library  system,  ASSET.  In  addition,  simulations 
written  in  Pascal  can  be  readily  translated  in  to  Ada. 

To  satisfy  the  requirements  of  the  project  a  combination  of  computer  simulation  and 
expert  system  features  needed  to  be  incorporated.  Activities  of  individuals  operating  in 
decentralized  decision  making  environments  require  the  expert  systems  approach.  The 
complex  nature  of  environments  in  which  organizational  decision  making  takes  place, 
and  the  equally  complex  characterizations  of  key  organizational  variables  (such  as 
communications  networks'  shapes,  reliabilities,  utilization  and  modification)  require 
advanced  computer  simulation  techniques.  A  discrete  event  simulation  is  appropriate 
for  these  purposes. 


observed  in  prior  studies.  Moreover,  some  have  successfully  predicted  the  behavior  of  subjects  in 
later  experiments.  Appendix  2  lists  some  typical  simulations  at  each  level  of  behavior. 

As  is  usual  with  computer  systems,  however,  the  name  ICARUS  could  be  interpreted  as  an  acronym, 
the  Iowa  Crisis  And  Response  User  Simulation. 
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Traditional  expert  systems  normally  consist  of  two  components,  a  database  and  an 
inference  engine  containing  a  set  of  rules.  The  first  contains  an  individual's  knowledge 
and  the  second  uses  that  to  generate  responses  to  new  data.  Knowledge  may  include 
not  only  awareness  of  the  current  state  of  the  organization,  but  who  knows  or  believes 
various  characterizations  of  the  current  situation.  The  decision  rules  can  also  be  quite 
complex,  for  example  telling  simulated  participants  how  to  decide  who  to  tell  what 
information  to  and  when. 

Discrete  event  simulation  requires  that  a  system  be  characterized  as  a  finite  state.  For 
example,  a  finite  state  would  specify  each  person’s  current  knowledge  as  well  as  her 
current  morale  and  stress.  Knowledge  would  be  about  the  organization's  environment 
and  current  condition,  others'  preferences/priorities,  knowledge,  and  premises,  one’s 
own  performance,  and  the  likely  consequences  of  alternative  responses  the  existing 
situation.  In  addition,  discrete  event  simulation  provides  for  instantaneous  changes  in 
states,  that  is  events.  They  can  include  acts  like  communication  and/or  distortion  of 
information,  choices,  restructuring  network,  and  the  loss  of  an  organizational  unit. 
Discrete  event  simulators  can  also  arrange  to  have  a  history  of  events  programmed  to 
affect  future  events. 

An  ICARUS  model  consists  of  the  following  components:^®  A  set  of  variables,  a 
geographic  model,  and  a  list  of  people.  The  variables  describe  observable  features  of 
the  real  world.  People  may  observe  the  values  of  variables,  people  may  change 
variables,  and  people  may  hold  possibly  conflicting  beliefs  about  variables. 

The  geographic  model  describes  the  places  where  people  may  go.  At  any  instant,  each 
person  is  in  some  place,  and  one  of  the  actions  people  may  take  is  to  move  to  another 
place.  Geographic  constraints  determine  how  long  it  takes  to  get  from  one  place  to 
another,  and  they  determine  which  moves  are  legal. 

Each  person  in  an  ICARUS  model  has  a  place,  a  set  of  beliefs  about  the  variables  of 
the  model,  and  a  set  of  rules  that  relate  those  beliefs  to  the  actions  a  person  may  take. 
The  actions  a  simulated  person  may  take  include  inspecting  the  actual  value  of  a 
variable,  speaking  a  belief  about  a  variable,  changing  a  belief  about  a  variable,  and 
moving  to  a  different  place. 

For  the  purpose  of  examining  the  results  of  simulation,  an  ICARUS  person  may  also 
print  out  a  note  in  the  simulation  output.  In  addition,  there  is  a  trace  option  to  make  the 
output  include  all  actions  taken  by  an  ICARUS  person. 

A  Very  Very  Simple  Example®® 

More  detail  about  these  components  can  be  found  in  Appendix  7. 

It  is  extremely  important  to  keep  in  mind  that,  when  complete,  the  Icarus  language  will  support  vastly 
more  complex  situations  and  decision  systems  than  these  simple  examples  would  seem  to  imply.  By 
working  with  simple  examples  we  have  been  able  to  discover  and  solve  critical  problems  which  would  be 
much  more  difficult  to  find  and  understand  with  more  complex  examples. 
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Although  the  ICARUS  project  was  motivated  by  complex  crises  like  United  232's  (flight 
which  crash  landed  in  Sioux  City,  lA)  in  which  intelligent  people  collaborate  to  overcome 
extreme  crisis,  the  development  strategy  for  ICARUS  was  incremental.  The  following 
graduated  sequence  of  examples  suggests  the  procedure  used  to  develop  and  refine 
ICARUS.  The  strategy  was  to  build  simple  models  using  different  approaches  to  the 
relationship  between  the  parameters  of  the  model  and  the  likelihood  of  successful 
outcome.  Once  a  simple  model  exhausted  it’s  ability  to  help  improve  ICARUS  it  was 
discarded.  The  examples  introduce  the  ICARUS  language  and  should  give  some  hint 
of  its  possible  applications.'^'^ 

Consider  a  world  with  two  rooms,  call  them  Here  and  There.  The  following  ICARUS 
model  describes  the  behavior  of  Fred,  a  fairly  stupid  fellow  with  no  beliefs  who  spends 
his  life  shuttling  from  here  to  there: 

GEOGRAPHY 

Here 

There  0:3 
There 
Here  0:4 
PEOPLE 
Fred 

PLACE  =  Here 
RULES 

PLACE  =  Here  ->  SET  PLACE  =  There 
PLACE  =  There  ->  SET  PLACE  =  Here 
END 

END 

Throughout  this  model,  indenting  is  used  to  show  the  relationship  between  the  parts  of 
the  model.  For  example,  the  place  names  Here  and  There  are  indented  an  equal 
distance  under  the  keyword  GEOGRAPHY.  Under  each  of  place  name  in  the 
GEOGRAPHY  section  is  a  list  of  the  places  that  can  be  reached  from  that  place.  For 
example,  from  Here,  you  can  get  to  There  in  three  minutes,  and  from  There,  you  can 
get  to  Here  in  four  minutes. 

There  is  only  one  person  in  the  list  of  people,  Fred.  The  indented  list  under  Fred  is  a 
list  of  his  attributes,  including  his  initial  place  and  his  list  of  rules. 

Fred's  rule  list  is  fairly  simple.  If  he  finds  himself  in  the  place  called  Here,  he  moves  to 
the  place  called  There,  a  move  that  will  take  three  minutes.  Similarly,  if  he  finds  himself 
in  There,  he  moves  back  to  Here. 


The  ICARUS  Reference  Manual  may  be  obtained  upon  request  to  the  author. 


ARI  -  MDA-903-90-C-0154  -  Final  Report  Page  26 


This  isn't  a  very  interesting  model  for  a  number  of  reasons.  First,  if  it  is  used  as  input  to 
ICARUS,  there  will  be  no  output.  This  is  because  there  is  no  request  to  trace  all  of  the 
actions  of  Fred,  and  because  none  of  Fred's  rules  ask  for  any  output.  The  second 
shortcoming  is  that  the  model  never  terminates;  Fred  will  shuttle  between  Here  and 
There  forever.  Finally,  of  course,  the  model  is  fundamentally  uninteresting  because 
nothing  of  consequence  ever  happens  to  Fred  in  his  mindless  shuffle  between  Here 
and  There. 

A  Very  Simple  Example 

In  the  next  example,  our  simulated  person  Fred  has  a  more  complex  problem  to  solve; 
he  must  navigate  from  Here  to  There  through  a  connecting  room  called  Nowhere.  In 
addition,  we  will  make  the  model  terminate,  and  we  will  make  it  output  a  trace  of  all 
actions  taken  by  Fred. 

VARIABLES 
Goal  =  Nowhere 
GEOGRAPHY 
Olympus 
Here 

Nowhere  0:1 
Nowhere 
Here  0:1 
There  0:2 
There 

Nowhere  0:3 
PEOPLE 
Chronos 

PLACE  =  Olympus 
RULES 

->  DELAY  =  0:20:30  TERMINATE 
END 
Fred 
TRACE 
PLACE  =  Here 
BELIEFS 
Goal  =  There 
RULES 

Goal  =  'There  AND  NOT  PLACE  =  There  ->  SET  PLACE  =  There 
PLACE  =  There  ->  BELIEVE  Goal  =  'Here 
Goal  =  'Here  AND  NOT  PLACE  =  Here  ->  SET  PLACE  =  Here 
PLACE  =  Here  ->  BELIEVE  Goal  =  'There 
END 

END 
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This  model  terminates  because  there  is  a  person  named  Chronos  in  a  place  called 
Olympus  who's  only  role  is  to  end  the  simulation  after  twenty  minutes  and  thirty 
seconds.  The  only  reason  for  the  place  called  Olympus  is  to  provide  a  place  for 
Chronos  to  work  from.  Olympus  is  an  unreachable  place,  and  Chronos  doesn't  interact 
in  any  way  with  Fred. 

In  more  interesting  models,  the  occupants  of  Olympus  can  be  given  more  work  to  do. 
For  example,  they  can  force  a  variety  of  trials  on  the  simulated  people  by  abruptly 
changing  the  values  of  variables,  and  they  can  be  used  to  enforce  the  "rules  of  nature" 
by  adjusting  the  values  of  variables  in  response  to  actions  taken  by  the  simulated 
people. 

The  geography  of  this  model  now  has  a  place  called  Nowhere  which  is  between  Here 
and  There.  The  total  distance  from  Here  to  There  is  still  three  minutes,  and  the  total 
distance  from  There  to  Here  is  still  four  minutes.  Now,  however,  there  is  no  direct  path 
from  one  to  another.  If  Fred  finds  himself  in  Nowhere,  there  are  two  places  he  could  go 
next.  In  order  to  preserve  the  pattern  of  behavior  from  the  previous  model,  Fred  needs 
to  remember  his  goal  as  he  enters  Nowhere  so  he  can  know  which  exit  to  take.  To 
allow  for  this  very  simple  mental  state,  we  introduce  the  variable  called  Goal;  Fred's 
mental  state  is  represented  by  his  belief  about  this  variable.  Fred  never  actually  looks 
at  the  value  of  the  variable,  but  we  need  to  set  the  value  so  he  can  have  a  well  defined 
goal. 

Fred's  rules  in  this  model  can  be  paraphrased  as  follows:  If  Fred's  goal  is  There  and  he 
finds  himself  not  to  be  There,  then  he  tries  to  move  There.  On  arriving  There,  he  sets 
his  personal  belief  about  his  goal  to  Here.  Another  rule  will  then  drive  him  to  Here,  at 
which  point,  he  sets  his  goal  to  There.  The  order  in  which  these  rules  are  stated  is 
irrelevant. 

Note  that  Fred's  rules  never  mention  the  intermediate  place  called  Nowhere.  If  he  is 
Here,  his  attempt  to  get  himself  to  There  will  land  him  in  Nowhere,  at  which  point,  he 
will  find  that  he  is  not  yet  There,  so  he  will  try  again  to  get  There. 

It  is  worth  noting  the  role  of  the  apostrophes  in  the  above  code.  These  are  used  as 
prefixes  on  the  symbols  Here  and  There  when  these  are  being  assigned  to  or 
compared  with  the  values  of  variables.  The  apostrophe  is  used  to  distinguish 
syntactically  between  constants  and  variable  names. 

Finally,  the  keyword  TRACE  under  Fred's  name  is  what  causes  the  simulation  to 
produce  output.  When  the  simulation  is  run,  the  following  output  is  the  result: 

Time  Person  Action 


0:01:00.00,  Fred  moves  to:  Nowhere 
0:03:00.00,  Fred  moves  to:  There 
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0:03:00.00,  Fred  believes:  Goal  =  Here 
0:06:00.00,  Fred  moves  to:  Nowhere 
0:07:00.00,  Fred  moves  to:  Here 
0:07:00.00,  Fred  believes:  Goal  =  There 
0:08:00.00,  Fred  moves  to:  Nowhere 
0:10:00.00,  Fred  moves  to:  There 
0:10:00.00,  Fred  believes:  Goal  =  Here 
0:13:00.00,  Fred  moves  to:  Nowhere 
0:14:00.00,  Fred  moves  to:  Here 
0:14:00.00,  Fred  believes:  Goal  =  There 
0:15:00.00,  Fred  moves  to:  Nowhere 
0:17:00.00,  Fred  moves  to:  There 
0:17:00.00,  Fred  believes:  Goal  =  Here 
0:20:00.00,  Fred  moves  to:  Nowhere 

0:20:30.00,  Chronos  terminates  simulation. 

Fred's  beliefs  at  termination  are: 

Goal  =  Here 

The  trace  of  activities  is  intended  to  be  subject  to  automated  analysis.  It 
consists  of  a  series  of  lines,  where  each  line  reports  one  action  -- 
at  TIME,  PERSON  said  VARIABLE  =  VALUE 
at  TIME,  PERSON  moved  to  PLACE 
at  TIME,  PERSON  set  VARIABLE  =  VALUE 
at  TIME,  PERSON  believed  VARIABLE  =  VALUE 
at  TIME,  PERSON  printed  NOTE 

Technical  Considerations 

The  rules  governing  a  simulated  person  are  only  evaluated  when  that  person  is  alerted. 
The  initial  alert  for  an  ICARUS  person  occurs  at  the  beginning  of  time.  After  that  point, 
the  person  alerts  himself  every  time  his  beliefs  change,  and  he  is  alerted  by  changes  in 
his  environment.  In  the  above  example,  for  example,  he  alerts  himself  when  he  enters 
a  new  place.  Although  it  is  not  illustrated  above,  an  ICARUS  person  will  also  be  alerted 
when  someone  in  the  same  place  speaks,  when  a  new  person  enters  the  place,  and 
when  an  alarm  goes  off. 

Each  rule  consists  of  two  parts,  a  guard  and  an  action.  These  are  separated  by  the 
implication  arrow  If  a  guard  is  found  to  be  true  when  a  person  is  alerted,  then  that 
rule  is  said  to  fire  and  the  resulting  action  is  scheduled.  An  empty  guard  is  always 
considered  to  be  true. 

When  a  rule  fires,  the  action  need  not  happen  immediately;  for  example,  moving  from 
one  place  to  another  takes  a  simulated  time  proportional  to  the  delay  indicated  in  the 
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GEOGRAPHY  section.  Additional  delay  can  be  specified  for  any  action  with  a  DELAY 
clause,  as  used  in  the  above  example  to  control  the  time  limit  that  Chronos  imposes  on 
the  model. 

Time  is  input  and  output  in  units  of  hours,  minutes,  and  seconds  (with  an  optional 
fractional  part  indicating  hundredths  of  a  second).  The  notation  used  is  conventional, 
with  colons  separating  the  fields. 

A  Simple  Example 

Building  on  the  above  model,  where  Fred  wanders  back  and  forth  between  Here  and 
There,  we  can  give  Fred  a  job  to  do.  Consider  the  following.  In  Here  is  a  meter 
showing  the  level  of  the  water  in  a  reservoir.  In  there  is  the  control  lever  for  the  outflow. 
Fred  must  try  to  hold  the  water  level  at  some  set  point.  To  model  this,  we  need  a 
physical  model: 

Flow  =  Inflow  -  Outflow 

The  total  flow  in  or  out  of  the  reservoir  is  equal  to  the  difference  between  the  flows  in 
and  out  of  the  reservoir.  Flow  can  be  measured  in  cubic  meters  per  second. 

dLevel/dt  =  Flow  /  Area 

The  rate  of  change  in  the  level  of  the  water  increases  with  increased  flow  and  it 
decreases  with  increased  reservoir  area. 

Outflow  =  Level  *  Valve 

The  flow  out  of  the  reservoir  increases  as  the  drain  valve  is  opened,  and  it  increases  as 
the  water  pressure  increases.  The  constant  of  proportionality  is  one  if  the  valve 
opening  is  measured  as  the  number  of  cubic  meters  per  second  that  flows  when  the 
reservoir  is  one  meter  deep. 

The  model  will  use  a  discrete  approximation  of  this  system,  where  Neptune,  another 
occupant  of  Olympus,  manages  the  rise  and  fall  of  the  waters  in  response  to  the  current 
inflow  and  valve  settings. 

VARIABLES 

Valve  =0.0  -- the  setting  of  the  drain  valve  (closed) 

Level  =  0.0  --  the  water  level,  in  meters 

Inflow  =1.0  --  the  rate  of  water  flow  in,  in  cubic  meters/second 

Area  =  1 0.0  --  the  area  of  the  reservoir  (a  constant) 

Goal  =  Nowhere 

GEOGRAPHY 
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Olympus 

Here 

Nowhere  0:1 
Nowhere 
Here  0:1 
There  0:2 
There 

Nowhere  0:3 
PEOPLE 
Chronos 

PLACE  =  Olympus 
RULES 

->  DELAY  =  0:20:30  TERMINATE 
END 

Neptune 

PLACE  =  Olympus 
RULES 

->  DELAY  =  0:0:1  SET  Level  =  OBSERVE  Level  + 

(OBSERVE  Area  * 

(OBSERVE  Inflow  - 

(OBSERVE  Valve  *  OBSERVE  Level))) 

->  DELAY  =  0:0:1  ALERT 
END 

Fred 
TRACE 
PLACE  =  Here 
BELIEFS 
Goal  =  There 
Level  =  0.0 
RULES 

Goal  =  There  AND  NOT  PLACE  =  There  ->  SET  PLACE  =  There 
PLACE  =  There  ->  BELIEVE  Goal  =  'Here 
Goal  =  'Here  AND  NOT  PLACE  =  Here  ->  SET  PLACE  =  Here 
PLACE  =  Here  ->  BELIEVE  Goal  =  'There 

PLACE  =  Here  ->  BELIEVE  Level  =  OBSERVE  Level 
PLACE  =  There  ->  ( 

Level  <  5  ->  SET  Valve  =  OBSERVE  Valve  +  0.1 
Level  >  6  AND  OBSERVE  Valve  >  0.0  -> 

SET  Valve  =  OBSERVE  Valve  -  0.1) 

END 

END 
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In  this  model,  Neptune  has  two  rules.  One  rule  causes  him  to  alert  himself  every 
second.  When  he  is  alerted,  he  reevaluates  all  of  his  rules.  The  other  rule  models  the 
laws  of  hydraulics  as  they  apply  to  the  reservoir  problem. 

Fred  has  two  new  rules,  one  of  which  applies  if  he  is  Here,  and  the  other  of  which 
applies  if  he  is  There.  When  he  is  Here,  he  updates  his  belief  about  the  level  of  the 
reservoir.  When  he  is  There,  he  either  opens  the  valve  a  bit,  closes  it  a  bit,  or  leaves  it 
alone. 

Note  that  in  these  rules,  there  is  a  clear  distinction  between  a  person's  beliefs  about  a 
variable  and  the  actual  value  of  the  variable.  Beliefs  are  set  with  the  BELIEVE 
keyword,  while  the  SET  keyword  changes  the  actual  values.  The  keyword  OBSERVE 
is  used  to  check  the  actual  value  of  a  variable,  while  variable  names  that  are  not 
qualified  refer  to  the  person's  belief. 

Modified  versions  of  this  example  almost  work,  but  CHRONOS  currently  goes  mad  with 
his  job  of  enforcing  the  laws  of  nature. 

Some  Limitations  of  Icarus 

ICARUS  has  a  number  of  additional  weaknesses.  The  above  examples  illustrate  some 
of  these.*^^  However,  the  most  important  weakness  is  that,  in  attempting  to  describe  a 
system  involving  both  people  and  a  physical  system,  two  different  approaches  to 
simulation  are  required,  expert  systems  and  discrete  event  simulation.  Discrete  event 
simulation  is  an  established  technique  for  modeling  systems  with  strict  attention  to 
temporal  constraints  like  those  experienced  in  crises.  It  is  commonly  used  in  areas 
such  as  logistics  models.  While  expert  systems  show  great  promise  for  modeling 
intelligent  decision  making,  our  attempt  to  incorporate  expert  systems  into  discrete 
event  models  in  ICARUS  has  demonstrated  that  fundamental  problems  arise  when  the 
two  approaches  are  combined.  Previous  work  involved  either  strong  simulation 
methods  combined  with  weak  expert  systems,  or  the  converse,  as  in  Saso  (Novick, 
1990).  These  weak  combinations  did  not  reveal  the  fundamental  problem  which  arises 
when  both  approaches  must  contribute  equally.  The  project  demonstrated  that, 
although  ICARUS  supports  an  event-centered  model  of  the  world,  a  model  which  may 
be  reasonably  well  suited  to  modeling  human  decision  making,  it  is  clearly  poorly  suited 
to  models  of  continuous  processes  such  as  the  filling  and  draining  of  a  reservoir. 
Despite  numerous  attempts  we  failed  to  learn  how  to  better  integrate  these  two  kinds  of 
models  satisfactorily. 

A  number  of  quite  critical  features  were  not  developed  as  a  result  of  the  time  consumed 
trying  to  solve  this  fundamental  problem.  For  example,  although  it  is  not  apparent  in 


In  discussing  ICARUS  weaknesses,  it  should  be  noted  that  the  current  ICARUS  implementation  is  so 
fragile  that  slight  variations  on  a  working  model  frequently  fail  to  work  properly.  This  discussion  focuses 
on  the  underlying  weaknesses  of  the  ICARUS  language  itself  and  does  not  focus  on  the  inadequacies  of 
the  current  implementation 
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any  of  the  simple  examples  given  here,  ICARUS  people  are  very  credulous.  No  matter 
how  complex  their  rule  base,  they  always  believe  everything  they  are  told.  Clearly,  it  is 
necessary  to  find  a  way  to  add  a  class  of  rules  that  determines  how  a  simulated  person 
interprets  what  they  are  told,  with  interpretations  based  on  what  they  believe  and  on 
who  is  saying  what. 

There  are  also  many  minor  annoyances  in  the  ICARUS  language  which  would  need  to 
be  resolved  once  the  major  problem  is  resolved  and  additional  critical  features  are 
successfully  added.  For  example,  it  is  annoying  to  have  to  list  all  subjects  about  which 
a  person  may  have  a  belief  as  model  variables.  Some  such  subjects  are  purely  local  to 
one  person,  for  example,  a  person's  goals.  We  need  a  way  to  let  a  person  have  such 
private  beliefs.  Another  minor  annoyance  is  the  need  to  initialize  the  value  of  every 
belief.  All  variables  have  initial  values,  and  it  would  be  nice  if  the  default  initial  value  of 
a  belief  was  an  accurate  reflection  of  the  initial  value  of  the  corresponding  variable. 

Conclusions  from  Phase  2 

Struggling  with  ICARUS  challenges  consumed  far  more  time  during  Phase  2  (and  in  to 
subsequent  years)  than  had  been  anticipated.  Ultimately  it  was  concluded  that  the 
effort  was  far  too  ambitious.  If  the  ICARUS  initiative  is  to  progress  beyond  this  point, 
the  kinds  of  resources  devoted  to  the  construction  of  SIMNET  (Alluisi,  1991 ;  Miller  and 
Thorpe,  1995)  will  be  required.  For  now  it  appears  that  simulation  approaches 
satisfying  the  purposes  of  this  project  will,  if  possible,  need  to  adapt  SIMNET  or  its 
successors  as  they  become  available  for  nonmilitary  uses. 

Work  with  ICARUS  reinforced  the  conclusion  that  many,  if  not  most,  of  the  central 
linkages  and  interactions  affecting  how  well  organizations  balance  routine  efficiencies 
and  crisis  readiness  remain  matters  of  speculation  more  than  scientific  theory  or 
findings.  As  would  be  expected  given  the  results  of  the  Phase  1  macro  analyses,  most 
ICARUS  design  options  had  to  be  resolved  by  judgments  rather  than  by  relying  on 
evidence  from  studies.  Theoretical  guidance  was  also  limited  because  so  few  theories 
address  intra-organizational  phenomena. 

Due  to  the  persistent  challenges  posed  by  ICARUS,  the  third  phase  of  the  project  was 
forced  to  develop  experiments  without  the  anticipated  guidance  of  results  from 
ICARUS-based  simulations. 

Simulators  as  Research  Beds 

In  the  last  decade  attempts  to  achieve  rigorous  tests  of  causal  propositions  in  complex 
settings  has  led  researchers  to  increasingly  rely  on  simulators.'^^  They  have  been 
encouraged  to  do  so  by  several  factors.  Among  the  most  important  is  the  fact  that 
organizations  are  increasingly  relying  on  distributed  decision  making  systems  in  which 


At  least  seven  simulators  have  been  used  in  experiments  on  distributed  decision  making.  For  a  review 
see  Weaver,  etal.,  (1995). 
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many  organizational  members  make  semi-autonomous  judgments  and  choices  and 
there  is  a  premium  on  adequate  coordination  of  those  decisions.  These  systems,  like 
simulators,  require  that  communications  be  conducted  over  electronic  media  rather  than 
face  to  face.  A  second  major  factor  encourage  use  of  simulators  is  advances  in 
computer  capabilities,  particularly  capabilities  created  by  linking  personal  computers  in 
local  area  networks.  A  third  factor  is  rapidly  falling  costs  of  computer  equipment.  A 
fourth  consideration  is  participants’  accelerating  familiarity  with,  and  interest  in, 
computer-based  tasks  (including  computer  games). 

Simulators  are  usually  designed  with  specific  research  hypotheses  in  mind.  For 
example,  just  as  Carley,  et  al.,  (1992)  extended  SOAR  (Laird,  et  al.,  1987)  from  a  focus 
on  individual  to  multi-person)  cognition  processes  in  their  PLURAL  SOAR  simulation, 
TIDE^  (Hollenbeck,  et  al.,  1995)  was  developed  to  extend  Brunswick’s  (1955)  lens 
model  paradigm  for  individual  judgment  to  distributed  decision  maker’s  assessments  in 
hierarchical  command  and  control  situations.  C.l.T.I.E.S.  is  an  experimental  vehicle 
designed  to  learn  how  two  interdependent  teams  learn  to  recognize  the  nature  of  civic 
emergencies  (Wellens  and  Ergener,  1988).  Because  the  main  goal  of  the  present 
project  was  to  improve  understanding  of  organizational  decision  processes,  an 
organizational  simulator,  ALCO,  was  employed  as  an  experimental  test  bed. 

ALCO 

ALCO  was  created  at  the  University  of  Iowa  from  1985  to  1987  relying  on  proprietary 
hardware  and  software  from  UNISYS.  From  1990  through  1993  it  was  refined, 
converted  to  Turbo  Pascal  in  a  DOS  IBM  personal  computer,  local  area  network 
environment,  and  transported  to  other  universities.'’^ 

Comparisons  of  ALCO  to  Other  Simulators 

In  some  respects  ALCO  resembles  SYNWORK1  (Elsmore,  1994).  Both  are  “man  in  the 
loop”  simulators  which  require  effective  individual  performance  by  human  participants. 
To  achieve  effective  performance  both  require  monitoring  complex  activities,  arithmetic 
reasoning,  and  memory.  Both  provide  feedback  and  the  difficulty  of  tasks  can  be  varied 
in  both  simulators.  However,  SYNWORK1  was  not  designed  for  exploring  group 
performance;  unlike  ALCO  it  focuses  on  individual  skills  and  performance  under  trying 
conditions  (such  as  in  sleep  deprivation  research). 

As  noted  previously,  ALCO,  unlike  ICARUS,  DISCUSS  (Stasser,  1981,  1988),  PLURAL 
SOAR,  and  ELM  (Carley,  1991;  Carley  and  Prietula,  1994),  is  a  “man  in  the  loop” 
organizational  simulator  rather  than  a  pure  simulation  model  lacking  actual  human 
inputs  to  the  system..  Like  many  simulations  (e.g.,  the  Garbage  Can  models  of  Cohen, 
March,  and  Olsen,  1972;  Anderson  and  Fischer,  1986;  Carley,  1986;  Masuch,  and 
LaPotin,  1989),  ALCO  is  concerned  with  issues  of  organizational  performance  and 


^  The  ALCO  programs  and  Manual  can  be  obtained  from  the  author.  However,  technical  support  is  very 
limited. 
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adaptation.  Unlike  the  Garbage  Can  models,  and  like  ELM,  ALCO  incorporates 
individual,  as  well  as  organizational  and  environmental,  variables. 

ALCO  was  inspired  by  many  of  the  same  considerations  that  led  to  the  development  of 
Distributed  Interactive  Simulation  -  DIS  standards  (Fletcher,  1994;  Fitzsimmons  and 
Fletcher,  1995),  SIMNET  (Alluisi,  1991;  Miller  and  Thorpe,  1995)  and  TIDE^ 

(Hollenbeck,  at  al.,  1995).  For  example,  all  three  simulators  require  at  least  two  human 
actors  to  make  numerous  decisions  of  many  kinds.  However,  unlike  DIS,  whose 
applications  emphasize  training,  ALCO  and  TIDE^  were  motivated  by  the  desire  to 
conduct  research  capable  of  strong  inference  about  the  systemic  effects  of  distributed 
decision  makers,  distributed  and  imperfect  information,  changes  in  communications 
links’  structures  and  quality,  and  numerous  other  properties  of  real  organizations  on 
decision  processes  and  their  outcomes  (such  as  accuracy,  satisfaction,  cohesion,  and 
learning).'^'*  ALCO  is  adaptable  to  tactical  military  applications  (DIS)  or  military 
command  and  control  (e.g.,  TIDE^^  settings  but  it  is  not  limited  to  military  contexts.  All 
three  simulators  emphasize  coordination  of  autonomous  agents  who  must  achieve 
success  in  complex  environments  to  survive.  Unlike  TIDE  ,  both  DIS  and  ALCO  require 
shared  views  among  agents  to  achieve  this  coordination.  Both  TIDE^  and  ALCO 
provide  each  agent  with  unique  environmental  data,  but  ALCO  and  DIS  often  provide 
overlapping  data  as  well.  Unlike  TIDE^  and  DIS,  ALCO  does  not  emphasize 
geographic  information  as  it  was  not  developed  to  simulate  battlefield  task  conditions 
(although  it  can  use  battlefield  interpretations  of  its  parameters). 

ALCO  Decision  Tasks 

A  central  feature  of  work  group  and  organizational  life,  one  not  included  in  most 
simulations  or  simulators  (e.g.,  Tide^),  is  that  many  roles  are  required  of  each 
participant.  In  the  context  of  distributed  decision  making,  most  members  must  be 
capable  of  making  decisions  which  can  be  defined  by  many  task  typologies.  Both 
unitary  (where  judgments  must  be  made  by  all  members  acting  as  a  whole  -  Laughlin 
and  Branch,  1972)  and  divisible  (where  each  member  can  make  separate  judgments 
which  contribute  to  collective  performance)  are  frequently  encountered.  Both  simple 
and  complex  judgments  (Shaw,  1964)  are  required.  At  times  disjunctive  decisions 
(where  collective  performance  hinges  on  the  resources  and  performance  of  the  most 
competent  member  -  Steiner,  1972)  are  required.  Other  situations  are  conjunctive 
(performance  hinging  on  the  poorest  individual  judgments  or  decisions).  Yet  others  are 
additive  (each  member’s  assessments  or  choices  are  summed  in  some  fashion).  Some 
stages  of  a  decision  process  emphasize  creativity,  others  logical  ability,  and  still  others 
social  persuasion  or  meshing  of  personalities  (Driskell  et  al.,  1989).  Some  problem 
solving  situations  are  of  the  “eureka”  type,  where  the  solution  or  answer  is  inescapably 
optimal;  but  most  decisions  contain  options  that  cannot  be  so  clearly  distinguished  in 
terms  of  their  relative  merit.  Some  decisions  occur  where  members  share  mental 


^  Of  course  ALCO,  like  DIS,  can  be  used  for  training.  However,  ALCO  was  intended  primarily  to  train 
administrators  (managers  and  executives)  rather  than  battlefield  commanders. 
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models  of  the  situation  and/or  shared  judgment  policies.  Other  decisions  arise  in 
contexts  of  cognitive  conflict  (Rohrbaugh,  1979)  or  dissimilar  images  of  the  situation. 

Crises  can  be  created  using  a  variety  of  ALCO  options.  A  few  are  noted  here.  One  is 
to  degrade  communications  by  replacing  some  proportion  (selected  by  the 
experimenter)  of  message  contents  with  random  symbols.  Another  is  to  change  the 
relative  importance  of  each  member’s  performance  frequently  and/or  radically.  A  third 
is  to  provide  too  many  sources  of  information,  creating  potential  information  overload. 

Or  communications  structures  and/or  decision  processes  can  be  manipulated  to 
allocate  excessive  work  for  some  members  and  little  for  others.  Or  the  level  of 
performance  (e.g.,  accuracy  in  team  estimates)  required  to  receive  (any  given  level  of) 
new  resources  may  be  suddenly  raised,  as  when  an  organization’s  environment 
suddenly  becomes  more  hostile.  One  of  the  most  salient  ways  to  effect  a  crisis  is  to 
impose  very  high  penalties  on  participants  for  the  time  they  consume  to  perform  their 
many  tasks. 

ALCO  is  designed  to  permit  a  greater  variety  of  decision  task  types  to  be  explored  than 
most  simulators.  For  example,  whereas  TIDE^  employs  only  a  divisible  task,  ALCO  can 
use  both  divisible  and  unitary  decision  tasks.  Whereas  TIDE^  requires  each  member  to 
only  make  judgments,  ALCO  requires  both  judgments  and  arithmetic  tasks  to  be 
performed.  ALCO  can  confront  members  with  additive  or  non-additive  tasks.  ALCO 
tasks  can  be  made  to  be  conjunctive,  disjunctive  or  additive  and  can  vary  this  attribute 
for  each  trial  if  so  desired.  Most  importantly,  ALCO  members  can  be  given  the  ability  to 
reconfigure  their  organizational  structures  and  processes,  unlike  TIDE^,  DISCUSS,  and 
most  other  simulators. 

ALCO’s  basic  decision  tasks  are  of  two  kinds,  explicit  and  implicit.  Both  kinds  of  tasks 
must  be  performed  individually  by  each  member  as  well  as  collectively  through  dialogs. 
As  will  be  discussed  shortly,  there  are  two  major  implicit  tasks,  assessing  covariations, 
and  judging  the  probable  value  of  alternative  organizational  arrangements.  These 
tasks  are  performed  within  the  confines  of  several  organizational  systems.  Among  the 
most  important  are  the  communications  network,  decision  making  system,  resource 
allocation  system  (as  in  budgeting),  and  (potential)  impact  system.*^  Depending  on  the 
experimental  question,  the  organizational  systems  themselves  may  be,  at  least  partially, 
under  the  team’s  control. 

The  impact  system  reflects  the  importance  of  each  member’s  personal  performance  to 
the  organization.  In  ALCO,  as  in  most  organizations,  some  members’  performances 
are  usually  more  critical  to  the  organization’s  performance  than  are  other  members’ 
performances.  Also,  as  in  many  organizations,  the  relative  importance  of  each 
member’s  performance  may  change  over  time  as  conditions  change.  Over  trials  an 
experimenter  may  create  any  desired  impact  structure.  For  example,  all  members  may 


These  structures  may  or  may  not  be  aligned.  For  example,  a  member  who  has  power  because  of  a 
central  position  in  the  communications  network  may  have  few  of  the  resources  needed  to  acquire 
adequate  information,  change  the  network,  or  communicate  clearly  over  existing  channels. 
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initially  be  equally  vital,  but  some  members  become  more  vital  than  others  over  time. 
Each  member’s  impact  is  implemented  by  using  an  equation  known  to  all  participants. 
The  equation  weights  each  unique  parameter’s  current  value  to  determine  each 
member’s  impact  (and  to  establish  the  team’s  best  possible  estimate). ALCO’s 
communications  and  decision  system  characteristics  will  be  easier  to  understand  once 
the  decision  making  tasks  members  must  perform  are  more  fully  described. 

ALCO  teams  or  organizations  may  have  as  many  as  12  members."*^  The  two  major 
explicit  tasks  are  to  formulate  individual  and  collective  estimates  of  several  parameters’ 
current  values.  First,  each  member  must  formulate  an  estimate  about  the  current  value 
of  a  parameter  which  is  the  unique  responsibility  of  the  member.  Then,  once  each 
member  has  formulated  a  judgment  about  his  or  her  unique  parameter’s  current  value, 
the  members  must  formulate  a  collective  estimate  or  forecast  using  the  estimated 
values  of  their  unique  parameters  as  effectively  as  possible. 

Each  member  (A,  B,  and  C  in  Figure  1  below)  has  one  or  more  potentially  perfect 
information  sources  (S^  through  Sg)  or  advisors  which  can  be  consulted.  One  of  the 
implicit  decision  tasks  for  each  member  is  to  determine  how  much  of  their  available 
resources  should  be  devoted  to  obtaining  advice  or  information.  The  experimenter 
determines  how  tightly  coupled  are  the  source’s  accuracy  and  the  resources  provided 
for  the  information  or  advice.  The  linkage  may  be  very  tight  for  some  sources  and  very 
loose  for  others,  varying  continuously  in  whatever  manner  the  experimenter  or  trainer 
elects.  Thus,  one  of  the  first  important  implicit  decision  tasks  involves  making 
judgments  about  covariation  between  a  given  source’s  accuracy  and  the  amount  of 
resources  to  be  “paid”  for  the  source’s  information  or  advice.  This  particular  task  can 
be  made  to  be  extremely  challenging  if  the  experimenter  elects  to  vary  the  actual 
amount  of  linkage  between  resource  “fee”  and  source  accuracy  over  trials.  Additional 
implicit  covariation  judgment  tasks  also  arise  with  respect  to  the  collective  judgments. 
These  are  described  shortly. 


Thus,  in  a  three  member  team  if  the  unique  parameters  for  members  A,  B  and  C  were  weighted  0,  2 
and  4  respectively,  C’s  impact  would  be  twice  B’s  and  A  would  have  no  impact.  In  addition,  the  best  team 
estimate  would  be  the  sum  of  twice  A’s  parameter’s  current  value  and  four  times  B’s  parameter’s  value. 
For  logistical  and  statistical  reasons  three  person  teams  were  used  in  this  project. 
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Figure  1 

Basic  ALCO  Task 


Si 


Where: 

Si  through  Se  are  advisors  or  information  sources  which  suggest  the  current  values  of  A,  B,  and  C 
respectively, 

A,  B  and  C  are  team  members’  estimates  of  three  different  parameters, 

T  is  the  team’s  estimate  of  an  environmental  parameter’s  value  which  is  a  function  of  the  true  values 
of  A,  B,  and  C, 

E  is  the  current  environmental  parameter’s  value,  and 

$  is  the  payoff,  replenishment  of  resources,  generated  by  the  accuracy  of  the  team’s  estimate. 

Numerous  interpretations  of  the  explicit  decision  tasks  can  be  used.  For  example,  in  a 
civil  defense  instantation  of  ALCO,  on  each  trial  one  member  could  be  required  to 
estimate  an  incoming  missile’s  speed  based  on  the  estimates  received  from  radar  and  a 
satellite.  Another  might  estimate  the  missile’s  direction  of  travel  based  upon  a  ground 
report  from  a  forward  observer  and  the  satellite.  A  third  member  might  estimate  the 
missile’s  warhead  size  based  on  intelligence  estimates.  The  first  implicit  covariation 
tasks  are  for  each  member  to  decide  the  amount  of  resources  to  expend  for  indicators, 
which  indicators  are  most  effective  (in  terms  of  accuracy  provided  relative  to  resources 
consumed),  and  whether  or  not  to  “purchase”  additional  data  from  any  of  the  available 
indicators.  The  experimenter  can  control  the  number  of  indicators  available  to  each 
participant,  whether  or  not  participants  share  any  indicator(s),  and  the  degree  of 
accuracy  each  indicator  will  provide  for  a  given  expenditure  of  resources. 

To  continue  the  civil  defense  example,  the  team’s  second  explicit  task  is  to  formulate  a 
team  judgment  or  estimate  (T  in  Figure  1).  In  the  example  it  could  be  to  estimate  the 
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risk  to  the  civilian  population,  in  terms  of  probable  numbers  of  lives  lost,  based  on  the 
members’  estimates  of  speed,  direction  and  warhead  size  of  the  missile.  To  achieve 
the  best  possible  team  estimate  (E  in  Figure  1)  each  member’s  unique  parameter  must 
be  assessed  with  perfect  accuracy,  then  those  values  must  be  aggregated  employing 
the  function  specified  for  that  trial.  The  more  accurate  the  members’  collective  (team) 
estimate  of  losses  (T)  the  more  effectively  the  population  can  be  protected,  hence  the 
more  replenishment  of  resources  ($  in  Figure  1)  available  to  the  team. 

Two  implicit  covariation  judgments  are  useful  at  this  point.  One  is  for  members  to 
assess  how  tightly  linked  are  accuracy  of  team  judgment  and  the  amount  of  subsequent 
replenishment.  In  formulating  this  judgment,  members  are  forming  impressions  of  their 
environment’s  munificence  -  the  amount  of  replenishment  likely  to  be  obtained  for  any 
given  level  of  accuracy.  The  other  implicit  covariation  task  arises  when  any 
replenishment  of  resources  is  distributed  to  individual  members  rather  than  being 
retained  in  a  central  pool.  In  such  cases  an  experimenter  may  have  elected  to  base 
distributions  on  a  (moving)  average  of  the  members’  individual  decision  performance, 
as  in  a  meritocracy,  divide  replenishments  equally  among  members,  or  maintain  a 
specified  allocation  (that  could  favor  some  members  over  others  irrespective  of  criteria 
related  to  the  members’  behaviors).  Thus,  each  member  faces  an  implicit  task  of 
judging  how  well  rewarded  is  high  quality  individual  decision  making,  yet  another 
covariation  assessment. 

Numerous  judgments  must  be  made  in  each  trial.  Not  only  must  each  participant 
decide  how  much,  if  any,  to  spend  for  information  or  advice,  but  they  must  also  decide: 
whether  to  accept  or  revise  the  recommended  estimates  they  obtain  from  their  sources, 
the  reasons  for  their  personal  and  the  teams’  performance  in  previous  periods,  whether 
it  would  be  appropriate  to  ask  others  for  information  and/or  resources  and/or  help,  if 
they  should  interrupt  their  current  activities  to  respond  to  others,  the  potential  value  of 
checking  on  others’  past  behaviors,  if  it  is  necessary  to  open  new  channels  of 
communication,  and  when  to  suggest  changes  in  procedures  or  team  estimates.  And 
these  are  but  a  few  of  the  choices,  judgments  and  forecasts  involved  in  every  period. 

Recall  that  the  second  important,  but  implicit,  ALCO  decision  task  is  to  assess  the 
probable  value  of  alternative  organizational  processes  and  systems.'*®  Irrespective  of 
the  decision  system  (e.g.,  dictator,  majority,  consensus,  etc.)  required  by  the 
experimenter,  important  but  implicit  communication  decisions  are  required.  Two  major, 
often  necessary,  implicit  communications  decisions  involve  network  structure  and 
message  redundancy. 

Structural  changes  in  networks  can  arise  in  two  ways.  Some  links  may  fall  in  to  disuse 
or  never  be  used  at  all.'*®  Or,  links  may  be  added.  For  example,  frequently  an 
experimenter  will  prescribe  a  communications  structure  among  members  (such  as  the 


Only  some  of  the  major  organizational  issues  involving  communications  structures  and  processes  will 
be  discussed  here  to  avoid  unnecessary  complexity. 

This  is  common  when  members  in  “all  channel”  networks  have  simple  or  routine  tasks. 
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classic  wheel  or  chain,  Shaw,  1964)  and  give  one  or  more  members  the  ability  to  add 
links.  Usually  links  can  be  added  only  by  consuming  resources,  and  not  all  links  “cost” 
the  same  amounts.  Thus,  some  member(s)  must  judge  which  new  links,  if  any,  would 
be  cost  effective.  Where  two  or  more  members  with  individual  resources  can  add  the 
same  link,  possible  conflicts  of  interest  may  arise  as  each  waits  for  the  other  to  act  in 
order  to  save  their  personal  resources.  Thus,  ALCO  can  create  free  ridership  dilemmas 
for  teams  to  resolve  if  they  are  to  achieve  adequate  performance  (Dawes  et  al.,  1986; 
Samuelson  and  Messick,  1986). 

Redundancy  in  the  content  of  messages  can  be  helpful  because  each  ALCO  link  may 
conduct  information  imperfectly,  some  links  more  so  than  others,  and  the  amount  of  a 
link’s  imperfection  may  be  varied  over  trials  in  a  planned  manner.®^  Where  noise,  in 
the  form  of  random  symbols  replacing  letters  or  numbers,  is  present  one  or  more  of  the 
members  may  be  authorized  by  the  experimenter  to  use  resources  to  increase  a  link’s 
fidelity.  If  none  of  the  members  have  that  ability,  or  those  that  do  have  it  elect  not  to 
use  it,  “noise”  in  a  link  may  cause  delay  or  misinterpretation  of  messages.  Two 
additional  ways  to  overcome  low  fidelity,  which  teams  may  not  discover,  are  to 
introduce  codes  and/or  increase  redundancies  within  a  message.  Thus,  ALCO 
challenges  members’  ability  to  innovate.  This  challenge  is  of  greatest  importance  in  the 
kinds  of  communications  crises  studied  in  this  project,  where  delays  in  innovation  can 
cripple  collective  decision  making. 

In  some  ALCO  setups  members  may  elect  to  alter  their  decision  processes.  For 
example,  teams  required  to  achieve  unanimity  may  decide  to  designate  one  member  to 
determine  the  team’s  estimate  and  then  communicate  that  value  to  every  other  member 
so  that  it  becomes  every  member’s  estimate.  If  so,  the  team’s  decision  system  is 
effectively  a  “dictatorship.”  More  often,  when  using  decision  systems  like  majority  vote, 
consensus,  and  plurality,  each  member  communicates  -  to  whatever  other  members 
they  can,  or  elect  to,  address  -  a  personal  estimate  of  their  unique  parameter’s  value, 
and  (later)  their  individual  recommendation  for  the  team’s  estimate  (e.g.,  of  the  number 
of  lives  likely  to  be  lost). 

It  is  to  be  expected  that  organizational  properties  will  often  have  interactive  effects.  For 
example,  the  current  communications  structure  and  decision  rule  both  affect  the 
number  and  routing  of  messages  flowing  through  the  network  as  well  as  each 
member’s  workload.  Dictatorship  usually  results  in  fewer  messages  but  centralizes  the 
information  processing  in  one  member  whereas  consensus  distributes  more  data 
through  more  messages  but  permits  debate  and  other  forms  of  decentralization  in  the 
team’s  decision  process.  Of  course  there  are  likely  to  be  asymmetries.  In  a  consensus 
decision  system  members  are  free  to  reallocate  aspects  of  the  decision  task  to  a 
greater  extent  than  is  possible  in  a  dictator  decision  system. 

ALCO  Variables 


Noise  in  a  channel  can  be  varied  from  none  to  100  percent  at  the  experimenter’s  discretion. 
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ALCO  permits  quick  and  easy  implementation  of  experimental  designs  (or  training 
situations)  that  confront  participants  with  tasks  whose  nature  cannot  be  easily 
anticipated  from  trial  to  trial.  Over  three  dozen  independent  variables,  (such  as  the  cost 
of  establishing  each  one  way  communications  link,  the  degree  to  which  each  member’s 
own  resources  devoted  to  search  are  linked  to  quality  of  information  produced  by 
search,  and  the  initial  or  possible  communications  networks)  can  be  manipulated  using 
a  series  of  menus.®^  As  has  been  suggested,  the  major  sets  of  independent  variables 
are  resources  (such  as  individual  funds  and  personal  advisors  for  each  member), 
rewards  (for  each  member  or  for  the  team  or  organization),  communications,  feedback, 
performance  evaluation,  and  decision  scheme  or  system  (e.g.,  dictator,  plurality, 
consensus). 

Most  of  the  independent  variables  can  be  used  not  only  as  inter-organizational 
manipulations,  but  as  intra-organizational  manipulations  as  well.  One  example  of  the 
latter  is  the  distribution  of  replenishments  over  trials.  Others  include  the  distribution  of 
members’  impact,  the  quality  of  data  flowing  through  each  communication  link,  the  utility 
of  information  a  decision  maker  obtains  from  his  or  her  sources,  and  the  availability  of 
any  one  source’s  information  to  each  member. 

More  than  twenty  dependent  measures  are  automatically  recorded.  They  include  such 
resource  choices  as  amount  used  by  each  member  for  acquiring  advice,  creating 
additional  communications  links,  and  giving  subsidies  to  other  members  or  units.  Each 
member’s  judgmental  accuracy  and  the  accuracy  of  the  team’s  estimate  are  recorded. 
Communications  dependent  measures  include  the  network  structure  at  each  moment, 
intensity  of  channel  usage  (each  way  in  a  link),  timing  of  message  transmissions,  and 
contents  of  each  message.  Performance  dependent  measures  (including  time  and 
resources  consumed,  length  of  subunit  and/or  organizational  lifetimes,  individual 
judgment  accuracy,  efficiency  of  team  performance)  are  obtained  for  individuals  or 
subunits  as  well  as  for  the  overall  organization.  All  recorded  data  are  time  stamped  to 
permit  calculations  of  latencies  and  lags. 

In  effect  ALCO  permits  members  of  a  simulated  organization,  acting  in  a  strong 
induction  (experimental)  setting,  to  not  only  perform  routine  and  adaptive  tasks,  but  to 
change  the  very  nature  of  their  own  organization  at  the  same  time.  This  is 
characteristic  of  the  actions  of  experienced  personnel  like  FAA  controllers  who  redesign 
complex  systems  “on  the  fly”  when  emergencies  arise  (McKenna,  1991).  If  ALCO  were 
a  flight  simulator  the  flight  crew  would  be  not  only  "flying  the  plane"  and  attending  to 
emergencies,  but  redesigning  the  aircraft  and/or  crew  responsibilities  at  the  same  time. 
For  example,  when  a  crisis  arises  because  a  communications  link  is  unexpectedly  lost 
participants  must  construct  a  remedy  relying  on  several  capabilities  (e.g.,  re-routing  via 


For  a  complete  listing  of  ALCO  variables  see  Appendix  8,  Additional  manipulations  can  be  achieved 
using  written  or  verbal  directions  that  supplement  ALCO's  independent  variables.  For  example,  each 
team  may  be  told  it  is  in  a  tournament  in  which  the  members'  pay  for  participating  depends  upon  how  well 
they  perform  compared  to  other  teams. 
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more  costly  paths,  greater  reliance  on  standard  operating  procedures  and  terminology, 
decentralization)  to  retain  adequate  coordination. 

There  are  many  other  features  which  make  ALCO  a  useful  experimental  vehicle.  The 
multiple  tasks  involve  both  social  and  organizational.  Many  of  the  variables  which  can 
be  manipulated  or  measured  are  at  the  organizational  rather  than  individual  level.  An 
extraordinary  number  of  possible  issues  and  designs  can  be  easily  created. 
Experimental  sessions  are  more  consistent  over  time  because  the  subjects’  situation  is 
automated  rather  than  subject  to  experimenter  error.  Sessions  can  be  run  on  most 
existing  local  area  computer  networks,  even  if  the  computers  are  physically  dispersed. 
Because  the  system  is  computerized,  most  participants  become  deeply  involved. 

There  are  several  challenges  posed  for  researchers  when  using  ALCO.  The  first  is 
learning  how  to  use  the  system’s  two  key  programs  which  set  up  experiments  and  the 
third  program  which  runs  the  network.  Care  must  be  taken  to  assure  that  the  parameter 
values  which  are  chosen  are  both  internally  consistent®^  and  faithful  implementations  of 
theory,  the  conditions  in  other  studies,  or  the  values  found  in  actual  situations. 

Nonverbal  communication  and  behaviors  cannot  be  manipulated  nor  measured.  Most 
relationships  between  variables  have  been  specified  in  a  linear  manner  as  a  first 
approximation  to  situations  which  may  be  nonlinear.  Statistical  power  is  expensive 
because  each  observation  requires  at  least  three  adults  working  for  up  to  six  hours. 
Finally,  each  experimental  session  produces  large  quantities  of  data,  challenging  users 
to  devise  efficient  data  reduction  techniques. 

Based  on  all  the  materials  which  had  been  reviewed  in  Phase  1  of  this  project,  it  was 
determined  that  ALCO  experiments  should  focus  on  stress,  decision  systems,  and  type 
of  crisis  facing  the  organization.  The  following  section  serves  to  review  relevant  studies 
on  these  topics.  It  also  provides  an  example  of  a  propositional  inventory  of  the  kind 
discussed  in  Phase  1 . 

Organizational  Tasks,  Structures  and  Decision  Systems: 

In  the  context  of  managing  complex  systems,  Perrow  (1984)  noted  an  inherent 
contradiction  faced  by  most  organizations.  Managers  of  complex  systems  must  be  both 
highly  centralized  (to  coordinate  actions)  and  highly  decentralized  (to  react  in  timely 
fashion)  when  balancing  routine  and  readiness.  In  organization  theory  it  is  virtually  an 
axiom  that  organizations  require  different  decision  making  structures  to  respond 
effectively  to  different  environments  and  to  routine  versus  non-routine  situations 
(Duncan,  1973).  Units  in  which  effective  performance  requires  members  to  sustain  high 
performance  in  both  individual  and  collective  tasks  are  likely  not  to  perform  as  well  as 
those  where  tasks  can  be  structured  to  consist  of  essentially  autonomous  individual 
activities  or  primarily  collective  activities  (Wageman,  1995).  Yet,  as  Perrow  (1984) 
implied,  in  many  contexts  it  is  not  possible  to  create  and  sustain  such  a  neat  division. 


“  The  feasibility  of  different  combinations  of  parameter  settings  can  be  estimated  by  constructing 
spreadsheet  analogs  to  the  experimental  designs  which  are  being  considered. 
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When  the  same  unit  must  conduct  routine  decision  processes  while  constantly 
maintaining  readiness  to  cope  with  crises,  the  cognitive  capabilities  of  members  are 
only  one  variable  affecting  performance.  Another  set  of  variables  involve  the 
relationships,  exchanges,  and  interactions  among  members  (Foushee,  1984;  Foushee 
and  Helmreich,  1987).  The  speed  and  accuracy  with  which  changes  in  task 
requirements  are  noticed  and  interpreted  depend  on  the  nature  of  these  variables.  As 
one  researcher  noted, 

"[Team  and  ojrganizational  structure  is,  among  its  other  functions,  a  search 
heuristic.  The  quality  of  organizational  search  attained  depends  critically  on  the 
division  of  responsibility  and  attention  among  the  organization's  members,  on  the 
patterns  of  interactions  among  members....  In  effect,  the  structure  of  the 
organization,  its  formal  and  informal  channeling  of  interactions,  and  its  biasing 
incentives,  are  a  concrete  heuristic  for  searching  the  vast  space  of  logically  possible 
policies  [necessary  for  success]."  (Cohen,  1982) 

Complicating  the  analysis  is  the  fact  that  organizations  and  groups  have  many 
structures,  any  one  of  which  may  dominate  in  a  particular  context.  For  example,  in  one 
context  a  member  may  be  the  most  central  person  in  the  activated  communications 
network,  a  marginal  member  in  terms  of  status,  and  play  an  intermediate  role  in  a 
decision  scheme  (e.g,  an  interpreter  of  data  gathered  by  others  who  make  suggestions 
to  the  ultimate  decision  maker.  In  another  context  the  same  member  may  be  forced  to 
act  upon  local  information  and  only  afterward  inform  others.  Thus,  structural  factors 
both  affect  decision  processes  and  are  affected  by  previous  decision  activities. 

Typically  organizations  respond  to  changes  in  their  environments,  shifts  in  the  balance 
between  routine  and  crises,  new  technologies,  and  turnover  by  altering  one  or  more 
structures.  For  example,  in  an  aerospace  firm  experiencing  high  morale  but  low 
productivity  when  everyone  could  freely  communicate  with  everyone  else,  performance 
and  satisfaction  first  fell,  then  rose,  as  communications  were  first  centralized  (in  a  wheel 
configuration)  then  decentralized  (in  the  form  of  a  circle  -  Mears,  1974).  The  result  of 
attempts  to  reconcile  all  the  forces  affecting  routine  and  adaptive  performance  is,  in 
many  cases,  a  sequence  of  structures  which  are  neither  well  integrated  nor  suitable  for 
many  task  situations.  Referring  to  a  mapping  of  communications  linkages  between  16 
units  used  to  make  just  one  kind  of  decision,  observers  noted: 

"The  irony,  and  the  tragedy,  is  that  each  of  the  223  linkages  taken  by  itself 
makes  perfectly  good  sense.  Well-meaning,  rational  people  designed  each  link 
for  a  reason  that  made  sense  at  the  time.  The  ...  sad  fact  is  that  when  we  use 
this  diagram  in  presentations,  we  don't  draw  shouts  of  'Absurd.'  Instead  we  draw 
sighs,  [and]  nervous  laughter...."  (Peters  and  Waterman,  1982). 

It  is  little  wonder  that  organizations  evolve  structures  and  procedures  which  are  difficult 
to  defend.  Even  in  Carley  and  Prietula’s  (1994)  relatively  straightforward  ELM 
simulation  study,  the  best  balancing  of  efficiency  and  readiness  depended  in  a 
complicated  manner  on  the  relationship  between  the  organization’s  information 
structure  (how  relevant  information  was  initially  distributed  among  members)  and  the 


ARI  -  MDA-903-90-C-0154  -  Final  Report  Page  43 


organization’s  decision  making  structure.  Under  comparable  information  conditions 
centralized  decision  systems  never  performed  better  than  decentralized  teams  making 
collective  decisions.  However  centralized  decision  structures  performed  more  poorly  in 
crises  than  did  decentralized  teams  regardless  of  the  distribution  of  information  within 
the  organization.  When  relevant  information  was  given  to  all  organizational  members  or 
when  each  relevant  datum  was  unique  to  a  single  member  (as  in  this  experiment),  both 
crisis  and  non-crisis  performance  was  inferior  to  conditions  in  which  subsets  of  relevant 
information  was  shared  by  various  subsets  of  the  organization’s  members. 

Carley  and  Prietula  simulations  used  a  very  simple  decision  task.  Experimental 
research  has  usually  found  that  when  decision  making  or  problem  solving  tasks  are 
“complex,”  decentralized  systems  yield  better  results  than  centralized  systems.  For 
example,  decentralized  communications  networks  provided  better  responses  to 
complex  situations,  crises  and  problems,  and  were  more  satisfying  to  members,  than 
centralized  networks.  When  tasks  were  simple,  centralized  networks  were  more 
efficient  (Shaw,  1954a;  Guetzkowand  Simon,  1955;  Shaw,  1964;  Harshbarger,  1971; 
Snadowsky,  1972;  Sypher,  1977).  However,  most  experimental  studies  have,  in  fact, 
not  really  used  very  complex  tasks.  One  that  is  frequently  used  is  the  common  symbol 
task,  using  pure  colors  (e.g.,  of  marbles)  for  simple  conditions  and  blended  colors  for 
complex  conditions.  That  is  an  extraordinarily  simple  task  when  compared  to  the 
complexity  of  decision  tasks  outside  the  laboratory. 

The  complexity  of  the  decision  task  faced  by  actual  decision  makers  can  be  extreme. 
Consider,  for  example.  Xerox’s  decision  about  “the  office  of  the  future”  -  whether  or  not 
to  initiate  production  of  personal  computers,  laser  printers,  and  local  area  networks 
before  they  were  even  considered  by  other  firms  (Smith,  1988).  The  radically  new 
technologies  (most  of  which  were  very  different  from  familiar  copier  technology), 
magnitude  of  resources,  uncertainty  about  market  demand,  personal  ambitions,  clashes 
of  managerial  and  scientific  cultures,  and  the  distance  between  Rochester  and  Palo 
Alto  were  only  a  few  complications.  Complexity  is  even  more  an  issue  when  decision 
makers  are  attempting  to  make  sense  of  crises  like  forest  fires  (Mclean,  1992), 
managing  leaks  from  nuclear  power  plants  (Perrow,  1984),  coping  with  reactions  to 
product  tampering,  or  assessing  the  intentions  of  approaching  aircraft  (US  Congress 
House  and  Senate  Committees,  1988  and  1993;  Rogers,  1992). 

Task  complexity  in  these  kinds  of  decision  situations  can  result  from  many  causes:  the 
number  of  elements  which  must  be  integrated,  their  dispersion  (among  people, 
geographically,  or  over  time),  their  novelty,  demands  for  immediate  but  unproved 
responses,  and  numerous  other  factors.  Thus,  in  a  real  world  context,  it  is  apparent  that 
(even  maximum)  complexity  in  most  experimental  tasks  is  minimal.  As  such,  the 
following  summery  may  substantially  underestimate,  or  even  misrepresent,  the  true 
relationship  between  task  complexity  and  organizational  or  group  structure. 

Relationships  between  structure  and  decision  processes  may  not  be  so  straightforward 
(Wood,  1973)  as  have  been  suggested  by  experiments.  And,  overtime,  they  may  be 
inconsequential.  For  example,  in  one  study  differences  in  performance  due  to  different 
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structures  of  communications  networks  declined  with  experience  and  were  virtually 
eliminated  with  appropriate  reinforcement  (Burgess,  1968).  One  researcher  found  that 
whether  or  not  network  structures  are  congruent  with  members’  status  levels  may  affect 
performance  and  satisfaction  as  much  as  the  match  between  task  complexity  and 
communications  structure  (Moore,  et  al.,  1972).  In  a  recent  study  of  reactions  to 
complex  situations  it  was  reported  that  decision  quality  was  unaffected  by 
centralization,  threat,  extent  of  emphasis  on  the  importance  of  accuracy,  or  perceived 
time  pressure  (Turner,  1992). 

Task  performance  requirements  are  likely  to  affect  communications  patterns,  and  other 
indices  of  structure  or  process  within  units.  There  is  experimental  evidence  that 
centralization  of  decision  making,  control,  and  communications  is  likely  to  be  affected 
by  many  variables  which  define  task  complexity.  Among  them  are  the  amount  of 
tension  (Staw,  et  al.,  1981)  or  uncertainty  (Bourgeois,  et  al.,  1978)  felt  by  members,  the 
distribution  of  information  among  participants  (Shaw,  1954),  the  power  or  status 
distribution  of  members  (e.g.,  Hunwitz,  et  al.,  1989),  and  the  available  technology 
(Siegel,  et  al.,  1986).  As  noted  below,  tension  or  stress  is  likely  to  arise  from 
uncertainty  (Cohen,  1959),  time  pressure,  threats,  work  or  information  overloads,  and 
contagion  of  others’  fears,  among  many  possible  sources. 

Surprisingly,  one  recent  experiment  found  that  groups  in  a  high-uncertainty,  low-threat 
condition  expressed  the  greatest  amount  of  tension  and  developed  the  greatest  degree 
of  centralization  (Argote,  et  al.,  1989).  One  would  have  expected  tension  and 
centralization  to  be  greatest  under  the  high-uncertain  and  high  threat  condition.  The 
sensation  of  tension  or  stress  under  high  uncertainty  is  consistent  with  the  threat-rigidity 
thesis  (Staw,  et  al.,  1981)  and  the  widely  accepted  view  that  communications  (Kano, 
1977),  decision  making,  and  control  become  more  centralized  in  crisis  or  stressful 
conditions  (Hetzler,  1940;  Janis,  1954;  Korton,  1962;  Herman,  1963;  Holsti,  1971; 

Smart  and  Vertinsky,  1977;  Milburn,  et  al.,  1983). 

Consistent  with  the  threat-rigidity  thesis  (Staw,  et  al.  1981),  one  survey  of  managers 
and  executives  who  had  been  confronted  with  an  international  financial  crisis  found  that 
the  more  unmanageable  the  situation  appeared  to  be  the  greater  was  the  centralization 
of  the  firm’s  control  system,  and  the  poorer  were  attitudes  (Kuklan,  1988).  Similarly, 
some  observations  of  flight  crew  communications  in  stressful  or  crisis  conditions  also 
support  the  threat-rigidity  thesis.  Subordinates  more  often  withhold  information  or 
refrain  from  questioning  orders  or  observations  from  superiors  when  under  stress 
(Foushee  and  Helmreich,  1988).  And  they  are  more  willing  to  acquiesce  to  superiors 
(Hamblin,  1958;  Klein,  1976;  Worchel,  etal.,  1977;  Driskell  and  Salas,  1991a). 

Other  studies,  however,  indicate  these  patterns  may  not  be  very  general.  One 
suggests  that  stress  causes  both  high  and  low  status  members  to  become  more 
receptive  to  messages  from  other  members  (Lanzetta,  1955;  Torrance,  1967;  Driskell 
and  Salas,  1991a),  more  equal  participation  in  decision  processes  (Lanzetta,  1995), 
and  more  solicitation  of  subordinate  opinions  by  superiors  (Torrance,  1967).  Another 
study  reported  that  increased  time  pressure  did  not  cause  influence  to  shift  upward 


ARI  -  MDA-903-90-C-0154  -  Final  Report  Page  45 


despite  increased  stress.  Information  processing  by  the  group  did  fall  (Gladstein  and 
Reilly,  1985).  Thus,  in  general,  the  evidence  fails  to  conclusively  support  the  “threat- 
rigidity,”  or  centralization  arguments  as  generally  valid  propositions. 

Facing  these  cross  currents  of  arguments  and  evidence,  many  would  agree  with  Vroom 
(1973)  that  there  are  situations  where  a  leader  must  decide,  perhaps  after  consultation 
with  group  members,  and  other  situations  where  (at  least)  a  majority  of  the  group’s 
members  should  formulate  solutions,  make  estimates,  or  select  among  alternatives. 
However,  available  evidence  provides  no  consistent  basis  for  formulating  universal, 
conditional  rules  for  matching  decision  systems  to  different  varieties  of  complex 
situations.  In  fact,  it  could  be  argued  that  where  group  or  organizational  members  are 
dispersed,  have  unique  personal  information,  yet  must  achieve  a  coordinated  decision, 
judgments  or  choices  are  inevitably  decentralized  and  the  infrastructure  required  to 
centralize  them  is  not  only  costly  to  establish  and  maintain,  but  is  also  too  rigid  and 
insensitive  to  important  but  obscure  cues.  In  these  circumstances  reliance  on  a 
centralized  hierarchy  may  even  subordinate  those  at  the  top  to  the  wishes, 
interpretations,  or  errors  of  those  reporting  to  their  superiors.  For  example.  Captain 
Rogers  was  clearly  the  last  person  on  the  Vincennes  who  could  have  chosen  not  to 
launch  missiles  against  Iran  air.  However,  it  can  be  reasonably  argued  that  his  choices 
were  substantially  preordained,  subordinated  to  the  incorrect  judgment  of  his  radar 
interpreter,  junior  officers’  repetitions  of  incorrect  interpretations  of  events,  his  personal 
objectives,  and  his  mental  set  (Simon,  1945). 

Perhaps  in  recognition  of  the  reality  that  a  leader’s  decision  is  inevitably  collective  in 
some  substantive  sense,  there  is  little  evidence  about  the  conditions  under  which  an 
individual  leader’s  decisions  are  superior  to  alternative  decision  systems  (such  as 
requiring  a  majority  vote  to  select  among  alternatives).  Most  experiments,  inspired  by 
trial  jury  deliberations  and  decisions,  compare  the  behaviors  and  decisions  reached 
under  majority  and  unanimity  rules  imposed  on  the  group,  but  do  not  explicitly  consider 
leader  decision  making  as  an  option.  There  is  evidence  in  the  “jury”  experiments  that 
decision  rules  affect  decision  processes,  judgments  or  choices,  post-decision  opinions, 
and  satisfaction.  For  example,  during  decision  process  there  will  probably  be  more 
conflict  between  members  in  groups  using  a  majority  rule  than  in  groups  using  a 
unanimity  rule  (Falk,  1982).  Groups  using  a  majority  rule  are  less  likely  to  exhibit 
symptoms  of  groupthink  than  groups  using  unanimity  (Kameda  and  Sugimori  (1993). 
Less  centralized  rules  like  consensus  tend  to  produce  more  accurate  judgments  than 
more  centralized  rules  like  reliance  on  a  subset  of  members  to  make  the  decision 
(Holloman  and  Hendrick,  1972).  Majority  vote  decisions  are  likely  to  be  less  extreme 
than  unanimous  decisions  (Miller,  1985).  Members  of  groups  using  the  unanimity  rule 
are  likely  to  be  more  satisfied  than  those  using  the  majority  vote  rule  (Hart  and  Sung, 
1976;  Kaplan  and  Miller,  1987). 

Only  two  experiments  could  be  located  which  considered  a  leader’s  role  in  decision 
processes.  In  a  business  game  the  addition  of  a  formal  leader  to  student  teams  failed 
to  change  team  performance  (Remus  and  Edge,  1991).  However,  in  a  task  requiring 
prediction  of  the  order  in  which  jurors  in  the  film  12  Angry  Men  would  change  their  vote 
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form  guilty  to  not  guilty,  groups  relying  on  a  leader  were  less  accurate  than  groups 
using  either  consensus  or  majority  vote  decision  rules  (Holloman  and  Hendrick,  1972). 

It  should  be  noted  that,  although  not  every  potentially  relevant  study  has  been 
summarized  here,  the  total  effort  devoted  to  learning  the  implications  of  various 
decision  rules  is  remarkably  small.  Nor  were  there  any  studies  comparing  alternative 
group  decision  rules  for  different  types  of  decision  tasks  or  situations.  In  all  the  studies 
the  groups’  task  was  simple,  requiring  only  that  different  members’  recollections  or 
expectations  be  resolved  in  to  a  single  response.  Finally,  none  of  the  experiments 
comparing  various  group  structures,  decision  systems,  rules  or  schemes  involved 
threat,  time  pressure  or  other  sources  of  stress. 

Effects  of  Stress 

Concerns  (c.f.,  Deitz  and  Thomas,  1991)  about  the  impact  of  threats,  stress,  and 
anxiety  on  the  judgments  and  performance  of  teams  where  high  levels  of  reliability  and 
competence  are  required  are  understandable.  It  is  widely  believed  by  researchers 
(e.g.,  Yates,  1990,  376)  that,  in  general,  increasing  stress  may  improve  decision  making 
up  to  a  point,  after  which  it  becomes  dysfunctional  in  terms  of  its  effects  on  individual 
and  collective  performance.  However,  a  close  reading  of  research  on  the  effects  of 
anxiety  and  stress  indicates  that  this  “inverted  U”  hypothesis  can  be  misleading  for 
many  reasons. 

One  reason  is  that  there  is  no  evidence  that  any  two  of  many  kinds  of  stress  have  the 
same  effects.  For  example,  one  might  ask  whether  psychological  (self-reported  or 
perceived  stress,  Appley  and  Trumbull,  1967;  Janis  and  Mann,  1977;  Lazarus,  1966), 
task  (e.g.,  stakes  in  possible  decision  outcomes  -  Yates,  1990),  cognitive  (e.g., 
information  overload,  Hamilton,  1982),  occupational  (e.g.,  role  overload,  Beehr,  1987), 
organizational  (e.g.,  work  pace  -  Quick  and  Quick,  1984),  physiological  (e.g.,  increased 
blood  pressure  or  heat  rate,  Selye,  1956),  ambient  or  environmental  (e.g.,  noise  or  heat 
-  Lazarus,  1966;  Hamilton,  1979)  and  social  (e.g.,  fear  of  peer  rejection  -  Kaplan,  1983) 
stresses  have  the  same  effects  on  any  given  performance  criterion  (e.g.,  an  individual’s 
judgmental  calibration  or  discrimination  -  Yates,  1990).  In  a  similar  vein,  it  seems  likely 
that  the  stresses  experienced  prior  to  and  during  combat  are  unlike  those  experienced 
in  most  civilian  task  settings,  so  their  effects  on  performance  are  not  likely  to  be 
comparable  (Driskell  and  Salas,  1991b). 

A  second  reason  for  cautious  acceptance  of  the  “inverted  U”  hypothesis  is  that  there  is 
limited  evidence  that  each  type  of  stress  will  have  unique  effects.  For  example,  Wright 
(1974)  found  that  one  stressor,  time  pressure,  reduced  attention  was  evident  in  greater 
emphasis  on  possible  negative  outcomes.  But  reduced  attention  due  to  another 
stressor,  noise,  was  not  systematically  related  to  a  particular  attribute  of  the  information 
environment.  If  this  result  generalized  to  other  settings,  environmental/ambient  stress 
would  be  expected  to  result  in  a  less  biased  sampling  of  incoming  information  than 
would  increased  time  pressure.  If  so,  providing  general  training  for  coping  with  stress 
would  be  less  appropriate  than  developing  specific  coping  behaviors  for  each  type  of 
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stress.  For  example,  where  threats  can  arise  from  many  sources  and  time  pressure  is 
expected  to  be  extreme,  helping  decision  makers  avoid  excessive  or  even  exclusive 
emphasis  on  negative  attributes  of  the  crisis  would  be  essential. 

A  third  reason  for  caution  regarding  the  “inverted  U”  hypothesis  is  that,  despite  very 
many  studies  of  the  effects  of  various  kinds  of  anxiety  and  stress  (Appley  and  Trumbull, 
1986),  direct  evidence  regarding  its  impact  on  decision  processes  has  important 
limitations.  It  is  sparse  and  almost  exclusively  focused  in  individual  rather  than 
collective  decision  processes  (Morgan  and  Bowers,  1995). 

A  fourth  reason  is  that  the  evidence  about  the  effects  of  anxiety  and  stress  on  decision 
processes  is  not  only  limited  but  inconclusive.  After  carefully  reviewing  48  review  and 
empirical  articles  focusing  on  individual  judgment  and  decision  making,  articles 
published  between  1980  and  1989,  Mross  and  Hammond  (1990)  observed, 

“No  generalization  regarding  the  effects  of  stress  on  judgment  and  decision  making 
can  be  readily  justified  on  the  basis  of  [these]  articles....  It  has  not  been  clearly 
demonstrated  that  stress  impairs,  enhances,  or  has  no  effect  on  cognitive  activity.” 

Several  features  of  Mross  and  Hammond’s  review  encourage  even  more  caution  for 
those  concerned  with  group  or  other  collective  decisions.  First,  time  pressure  was  the 
source  of  stress  for  nearly  25  percent  of  the  articles;  physical  danger  was  a  stressor  in 
only  four  articles.  Of  the  four,  only  one  (Weltzman,  et  al.,  1971)  was  an  experiment 
indicating  any  adverse  effects;  a  narrowing  of  focus.  Second,  they  offered  no 
conclusion  about  the  effects  of  stress  on  group  or  organizational  decision  processes. 

Furthermore: 

“The  large  literature  regarding  the  effects  of  stress  ...  has  nearly  overlooked  the 
effects  on  team  processes  and  performance.”  (Morgan  and  Bowers,  1 995) 
“Organizational ...  research  [has]  been  too  little  concerned  with  organizational  and 
interpersonal  factors  that  might  serve  as  moderators,  buffers,  or  even  as  antidotes 
to  stresses  and  their  effects....  [E]mpirical  work  in  this  promising  area 
[organizational  stress]  is  almost  non-existent.”  (Kahn  and  Byosiere,  1990). 

What  guidance  can  the  limited  research  offer  those  concerned  with  balancing  efficiency 
and  readiness?  The  following  (tentative)  propositions  relating  to  stress  were  obtained 
from  the  Phase  1  macro  analysis. 

Individuals  who  exhibit  higher  test  anxiety  review  more  information  than  those  who 
whose  test  anxiety  scores  are  lower.  (Nichols-Hoppe  and  Beach,  1990). 
However,  note  Wright’s  (1974)  findings  regarding  effects  of  time  pressure  listed 
below. 

High  test  anxiety  individuals  also  review  more  previously  considered,  redundant 
information  (Nichols-Hoppe  and  Beach,  1990). 

The  greater  the  (test)  anxiety,  the  less  attention  is  focused  on  the  task  (Sarason, 
1990;  Wine,  1971). 
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Low  anxiety  persons  take  longer  to  make  judgments  than  high  anxiety  people 
(Singh,  Sharma,  1989). 

Low  anxiety  individuals  are  less  likely  to  conform  than  high  anxiety  individuals 
(Singh,  Sharma,  1989). 

The  greater  one’s  anxiety  the  more  cautious  the  interpretation  of  events;  i.e.,  the 
more  stringent  the  criterion  for  reporting  detection  of  a  signal  (in  a  signal 
detection  task)  (Geen,  1985). 

Low  test  anxiety  individuals  perform  selective  attention  tasks  better  than  high 
anxiety  individuals  (Upadhayay,  et  al.,  1985). 

In  selective  attention  tasks  people  experiencing  physiological  stress  (due  to  random 
shocks)  perform  more  poorly  than  those  not  under  stress  (Upadhayay,  et  al., 
1985). 

When  exposed  to  the  risk  of  being  shocked.  Type  A  individuals  will  exhibit  more 
physiological  stress  -  as  measured  by  heart  rate  -  than  Type  B  individuals  (Evans 
and  Fearn,  1985). 

Type  A  individuals  will  engage  in  more  active  monitoring  of  their  environment  when 
threatened  (with  being  shocked)  than  Type  B  individuals  (Evans  and  Fearn, 
1985). 

The  greater  the  time  pressure  the  fewer  attributes  of  options  are  systematically  used 
to  make  a  choice  (Wright,  1974).  Note  that  this  contrasts  with  the  anxiety  effects 
noted  previously. 

Also,  the  greater  the  time  pressure  the  more  attention  focuses  on  negative  attributes 
of  the  options  (Ben-Zur  and  Breznitz,  1981;  Wright,  1974). 

Multi-Attribute-Utility  decision  models  are  less  utilized  under  time  pressure  than 
without  time  pressure  (Zakay  and  Wooler,  1984). 

In  decisions  under  certainty,  the  greater  the  time  pressure  the  greater  the  reliance 
on  non-compensatory  heuristics  (Zakay,  1985).  Use  of  non-compensatory  rules 
and  confidence  are  positively  related.  Note  that  simulation  results  suggest  this 
will  not  reduce  the  quality  of  decisions  if  the  right  heuristics  (e.g..  Elimination  By 
Aspects,  Equiprobability)  are  used,  but  not  if  other  heuristics  are  invoked  (Payne, 
et  al,  1993,  143). 

Also  in  decisions  under  certainty,  the  more  time  pressure  the  more  teams  will  focus 
on  the  most  important  attributes  of  the  situation  (Ediand,  1994).Training 
improves  MAU  decision  model  utilization  only  in  absence  of  time  pressure 
(Zakay  and  Wooler,  1984). 

The  greater  the  time  pressure  the  less  predictable  the  judgment  or  choice  in  a 
complex  (curvilinear  SJT)  judgment  task.  That  is,  as  time  pressures  rise, 
decision  makers  continue  to  use  the  same  data  and  give  each  datum  the  same 
weight,  but  they  use  the  data  more  erratically.  In  simple  (linear)  tasks  time 
pressure  has  no  effect  on  cognitive  control  (Rothstein,  1986). 

The  more  time  pressure  a  group  or  team  experiences  the  less  information  it 
processes  or  exchanges  (Goldstein,  Reilly,  1985). 

The  extent  of  centralization  in  a  team  or  group  is  unaffected  by  time  pressure, 
contrary  to  the  “threat-rigidity”  literature  (Goldstein,  Reilly,  1985;  Staw,  et  al., 
1981). 
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There  are  no  differences  in  efficiency  between  groups  under  high  and  low  time 
pressure  (Isenberg,  1981). 

The  distribution  of  “air  time”  among  members  will  be  more  centralized  in  groups 
under  time  pressure  than  those  not  under  pressure  (Isenberg  (1981).  Thus, 
Communications  become  more  centralized  under  time  pressure. 

The  more  perceived  time  pressure  the  lower  the  number  of  plans  made  using 
moderately  complex  planning  schemes;  but  there  are  no  quantity  differences 
when  the  planning  scheme  is  simple  (Abualsamh,  et  al.,  1990). 

The  more  stress  from  environmental  noise  the  less  predictable  the  decision  maker’s 
evaluation  of  options  (Wright,  1974).  Thus  both  time  pressure  and 
environmental  stress  appear  to  cause  decision  makers  to  rely  less  consistently 
on  their  judgment  models  or  heuristics. 

Noise  and  task  overload  stress  increase  the  use  of  the  representativeness  heuristic 
(Shaham  et  al.,  1992). 

Neither  overload  nor  noise  affect  risk-aversion  in  evaluating  options  (Shaham  et  al., 
1992). 

When  in  noisy  or  crowded  conditions,  groups  with  predetermined  structures  perform 
more  poorly  than  those  without  a  previously  specified  structure  (Worchel  and 
Shackelord  1991). 

Classification  accuracy  can  be  maintained  over  a  wide  range  of  work  paces  (Coury, 
Drury,  1986). 

The  more  stress  the  greater  the  reliance  on  heuristics  (Schaeffer,  1989). 

The  greater  the  stress  the  greater  one’s  confidence  (Schaeffer,  1989). 

When  judgments  are  made  in  the  presence  of  others  dominant  or  habitual 
attentional  responses  occur  more  than  would  be  the  case  in  the  absence  of 
others;  the  presence  of  others  tends  to  reduce  even  more  attention  to  factors  a 
decision  maker  rarely  notices  when  not  under  stress  (Zajonc,  1965). 

The  greater  the  stress  the  more  subordinates  defer  (refrain  from  questioning  and 
hesitate  to  act  in  emergencies)  to  a  superior's  authority  and  interpretations  of 
events  (Foushee  and  Helmreich,  1988;  Driskell  and  Salas,  1991a). 

The  greater  the  stress  the  more  receptive  group  or  team  members  (both  high  and 
low  status)  are  to  task  information  from  others  (Lanzeta,  1955;  Torrance,  1967; 
Driskell  and  Salas,  1991a). 

The  more  stress  the  greater  the  tendency  to  offer  others  solutions  to  problems 
before  all  decision  alternatives  have  been  considered  (Keinan,  Friedland,  and 
Ben-Porath,  1987)  (Keinan,  1987). 

The  more  stress  the  less  systematically  alternatives  are  examined.  (And  the 
scanning  patterns  correlate  with  the  quality  of  solutions  to  problems.)  (Keinan, 
Friedland,  and  Ben-Porath,  1987;  Keinan,  1987). 

A  group’s  decision  quality  is  not  affected  by  perceived  time  pressure  or  perceived 
threat  (Turner,  1992). 

Time  pressure  increases  a  group’s  decision  making  speed  (Turner,  1992). 

Threats  cause  arousal,  and  arousal  is  contagious,  being  spread  by  nonverbal  cues 
in  either  face  to  face  or  distributed  groups  (Levy  and  Nail,  1993;  Hatfield,  et  al., 
1994).  Thus,  messages  containing  even  a  hint  of  worry  or  stress,  particularly  if 
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they  are  numerous  and  arrive  at  accelerating  rates  from  many  sources,  may  be 
expected  to  produce  a  vigorous  sensation  of  panic  or  crisis. 

External  threats  increase  team  or  group  cohesion  (Sherif  at  al.,  1961). 

External  threats  provoke  greater  attention  to,  reliance  on,  and  support  for  leaders’ 
and  influential  members’  behaviors  and  opinions,  increase  pressures  to  conform, 
encourage  more  consensus  seeking  (even  if  achieved  by  considering  less 
information),  and  foster  greater  centralization,  i.e.,  growing  control  of  those  with 
dominant  perspectives  over  other  members  (Staw,  et  al.,  1981). 

Although  not  adequately  documented,  it  has  been  argued,  using  social  facilitation 
(Zjonc,  1965)  reasoning,  that  performance  will  to  depend  on  whether  or  not 
dominant  responses  encouraged  by  threat  serve  to  improve  decision  making 
performance  (Staw,  et  al.,  1981). 

It  has  also  been  argued,  but  not  well  documented,  that: 

The  effects  of  threat  on  the  decision  making  unit  depend  on  the  source  of  threat. 
Internal  threats  reduce  cohesion,  encourage  less  respect  for  leaders  and 
influential  members,  and  induce  decentralization  or  even  conflict. 

External  threats  tend  to  cause  the  group  or  organization  to  become  more 
mechanical  in  nature  whereas  internal  threats  may  cause  them  to  become 
more  like  organisms  (Burns  and  Stalker,  1961). 

These  propostions,  while  not  an  absolutely  exhaustive  scan  of  all  empirical  findings 
relevant  to  the  effects  of  stressful  conditions  on  decision  processes, are  noteworthy  in 
several  respects.  Most  are  founded  on  a  single  study  using  decision  tasks  which  differ 
from  other  studies’  tasks  in  undefined  ways.  Replications  demonstrating  that  findings 
from  ad-hoc  groups  of  college  students  in  simple  task  settings  are  needed  to  establish 
the  propositions’  generality.  Another  observation  is  that  only  a  very  few  of  all  possible 
sources  of  stress  have  been  examined.  Yet  another  is  that  these  results  provide 
virtually  no  evidence  about  the  joint  effects  of  two  or  more  stressors  which  are 
experienced  simultaneously.  It  seems  reasonable  to  speculate  that  the  cumulative 
effects  of  multiple  stressors  is  not  additive,  but  multiplicative;  each  additional  source  of 
stress  accelerates  the  arrival  and  magnitude  of  dysfunctional  decision  making  practices. 
Overall,  studies  of  the  effects  of  various  sources  and  forms  of  stress  on  decision 
making  reinforce  the  reservations  expressed  by  Mross  and  Hammond  (1990)  and 
Morgan  and  Bowers  (1995)  noted  earlier. 

There  a  many  propositions  which  appear  to  have  received  no  empirical  attention  but 
which  have  potentially  important  implications.  For  example 

Units  achieving  very  high  levels  of  efficiency  on  routine  activities  develop  dominant 
habits  which  hinder  decision  making  in  crises. 

Organizations,  groups  or  teams  which  have  not  faced  crises  and  those  which  have 
coped  successfully  with  one  source  or  type  of  crisis  may  make  equally  good 
decisions. 

Training  teams  for  coping  for  crises  will  be  effective  to  the  extent  that: 


“  Franken  and  O’Neil  (1994)  is  one  study  which  was  not  available  to  us.  It  examined  the  effects  of 
anxiety  in  Navy  teams  on  their  performance. 
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Many  different  kinds  of  crises  are  imagined  when  developing  training  scenarios, 
and 

Patterns  of  information  associated  with  each  type  of  crisis  can  be  distinguished 
by  members  of  training  groups  (Klein,  1993),  and 
Members  are  able  to  generate  new  interpretations  of  information  patterns  when 
initial  interpretations  are  unproductive  (Gettys  and  Fisher,  1979;  Gettys  et  al., 
1987). 

This  review  of  some  of  the  research  which  is  most  relevant  to  this  project  should  have 
reinforced  the  concerns  noted  in  Phase  1.  Most  propositions  have  not  been  replicated 
and  have  been  tested  in  contexts  that  are  both  unique  and  far  less  complex  than  those 
decision  makers  face  every  day.  The  next  section  describes  how  ALCO  was  used  in  an 
attempt  to  resolve  some  of  the  limitations  of  the  literature  which  is  relevant  to  this 
project. 

Piloting  ALCO 

ALCO  experiments  were  conducted  in  relatively  uncharted  waters.  Given  the  limited 
resources  available  for  creating  personal  and  team  incentives,  great  effort  was  made  to 
assure  that  experiments  would  produce  useful  insights.  Thus,  prior  to  attempting  a 
complete  experiment,  a  series  of  pilot  partial  experiments  with  ALCO  were  conducted 
over  a  period  of  nearly  two  years.®"*  They  addressed  not  only  the  ALCO  challenges 
discussed  earlier,  but  also  those  difficulties  involved  in  any  experiment  requiring 
multiple  subjects  per  observation,  unknown  strength  of  effects  of  manipulations  of 
independent  variables,  and  similar  problems.  This  involved  too  many  obstacles  and 
false  starts  to  fully  describe  in  this  report.  Only  the  most  difficult  are  noted  here. 

During  piloting  it  was  discovered  that  student  subjects,  even  graduate  students, 
required  at  least  90  minutes  of  training  and  another  60  to  120  minutes  (depending  on 
the  complexity  of  the  simulation  situation)  to  become  proficient.  Although  most  subjects 
became  very  involved,  even  in  designs  without  financial  incentives,  feedback  indicated 
the  initial  learning  curve  was  steep  and  frequently  frustrating.®®  It  was  discovered  that 
subjects  became  much  more  patient  and  became  less  frustrated  when  they  were 
informed  at  the  outset  that  their  experience  would  be  much  like  flight  simulation,  and 
they  could  only  expect  to  fly  effectively  if  well  trained. 

Early  debugging  and  piloting  involved  undergraduates  because  they  are  routinely 
available  in  a  subject  pool.  This  proved  unworkable.  The  time  commitments  required 
of  them  in  ALCO  studies  were  longer  than  their  other  options  for  satisfying  the  pool 
requirement.  Those  who  did  sign  up  to  participate  often  did  so  to  assure  they  would 
have  access  to  some  study  satisfying  the  requirement.  When  other  options  arose  and 


These  constituted  partial  experiments  at  best.  Each  examined  a  different  design  question. 

“  Frustrations  for  researchers  and  subjects  were  greatest  when  the  system  collapsed.  This  was  a 
frequent  problem  when  the  computer  laboratory  was  moved  and  new  hardware  and  network  software  was 
installed,  necessitating  substantial  revisions  in  the  ALCO  software. 
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were  exercised  they  failed  to  appear  at  the  appointed  times®® .  Of  those  who  did 
participate  as  scheduled,  not  all  were  willing  to  exert  the  effort  required  to  become 
proficient.  These,  and  other  problems  thwarted  the  first  efforts  to  debug  our  research 
design.  It  was  concluded  that  only  subjects  who  volunteered  would  be  suitable  for 
projections  of  this  kind.  Further,  it  was  determined  that  only  MBA  students  would  be 
sought  as  they  are  more  mature  and  have  more  work  experience. 

One  pilot  study  demonstrated  that  MBA  participants  substantially  increased  their 
attentiveness  to  the  ALCO  situations  they  confronted  when  individual  and  team 
compensation  incentives  were  supplemented  with  competition  and  public  feedback 
about  all  teams’  performance.  As  a  result,  as  described  later,  the  major  study  mounted 
in  Phase  3  used  a  tournament  pay  system  coupled  with  public  posting  of  results.  The 
issues  and  procedures  in  that  experiment  are  described  in  the  following  sections. 

This  kind  of  study  has  been  criticized  by  those  who  emphasis  naturalistic  decision 
research  for  many  reasons  (e.g.,  Klein,  1993).  Objections  to  the  use  of  student 
subjects  who  are  placed  in  ad  hoc  teams  and  given  too  little  time  to  formulated  social 
systems  are  common.  However,  there  are  reasons  to  believe  that  many  laboratory 
studies  do  generalize  to  field  settings  (c.f.,  Locke,  1986;  Levin,  et  al.,  1983;  Driskell  and 
Salas,  1992).  Given  that  field  experiments  are  even  more  difficult  to  conduct  than 
laboratory  studies,  it  is  a  wiser  use  of  resources  to  begin  with  laboratory  experiments  of 
kind  reported  here. 

Design 

The  most  extensive  experiment  used  39  experienced  MBAs  randomly  assigned  to  13 
three  person  teams®^  in  a  2  x  2  within-team  design  (Table  2).  The  four  conditions  were 
designed  to  examine  the  effects  of  different  levels  of  crisis  severity  and  two  decision 
making  rules  or  systems.  In  the  “local,”  low  severity,  condition  only  one  member’s 
advisor  unexpectedly  (and  without  warning)  became  far  less  accurate  than  previously. 

In  the  “global,”  high  severity,  case  all  members’  advisors  became  less  accurate  at  the 
same  time,  again  without  warning.  In  the  “dictator”  or  leader  decision  system  one 
member’s  estimate  of  the  environmental  parameter’s  (E’s)  value  constituted  the  team’s 
response  (T).  When  it  was  submitted  by  the  leader  the  trial  ended  and  a  new  one 
began.  In  the  consensus,  unanimity,  decision  system  every  member  had  to  agree  on 
the  same  value  of  the  environmental  parameter’s  estimate  (T)  to  conclude  a  trial.  Major 
dependent  variables  included  individual  and  organizational  performance, 
communications,  commitment  (cohesion),  and  stress. 

Two  sessions  were  conducted  for  all  participants.  The  first  trained  them  in  ALCO  and 
collected  personal  data.  They  learned,  for  example,  howto:  open  channels  of 

It  was  later  discovered  that  the  prevailing  norm  was  that  signing  up  to  participate  in  a  study  did  not 
constitute  a  binding  promise  to  appear.  This  was  even  true  in  many  instances  when  phone  call  reminders 
had  been  received  within  the  previous  24  hours. 

Groups,  teams,  and  organizations  differ  on  many  dimensions  (Swezey  and  Salas,  1992).  The  results  of 
this  experiment  cannot  be  generalized  to  organizational  settings. 
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communication,  send,  receive,  and  review  messages,  acquire  advice  from  their 
sources,  transfer  resources  to  other  members,  review  others’  past  activities,  and 
interpret  personal  and  team  feedback.  The  second,  “baseline”  session,  provided  an 
opportunity  to  become  proficient  in  the  use  of  those  skills  and  explore  what  could  be 
achieved  when  the  skills  were  used  effectively.  Thus,  by  the  time  participants 
encountered  the  first  experimental  session  they  had  achieved  reasonable  proficiency  in 
operating  the  system. 

As  indicated  in  Table  2,  data  were  collected  in  four  experimental  sessions  each 
containing  three  trial  blocks.  The  first  and  third  trial  blocks  within  each  session  were 
non-crisis  situations  involving  five  and  four  trials  or  periods  respectively.  However,  in 
the  second  block  either  one  (local  -  low  severity)  or  all  (global  or  high  severity) 
members’  advisors  become  less  useful  for  six  trials.  (This  change  was  unannounced, 
but  subjects  learned  to  anticipate  it  after  a  few  sessions.)  Thus,  teams  had 
opportunities  to  develop  high  levels  of  efficiency  in  routine  operations  before 
confronting  challenging  conditions  which  required  a  “shift  of  cognitive  gears”  (Louis  and 
Sutton,  1991),  innovation  in  individual  task  performance,  and  changes  in  the  team’s 
patterns  of  interaction  to  respond  well  to  the  crises. 
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Table  2 

Experimental  Design 


Order 

Order 

Order 

Order 
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Crisis 

Periods 
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Crisis 
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DecRule 

Crisis 
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Majority 

None 

10 

3 

Majority 

None 

10 

3 

Majority 

None 

10 

3 

Majority 
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10 

3 
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10 

3 

Majority 
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10 

3 

Majority 
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10 

3 
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10 

3 
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3 

3 
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sus 
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sus 
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6-12 

Dictator 
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13-16 

sus 
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Dictator 
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sus 
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None 
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sus 
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3 
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3 

Consen¬ 

None 

1-5 

Dictator 

None 

1  -5 
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6-12 
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6-12 
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None 

13-16 
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Each  team  experienced  all  four  experimental  conditions  in  one  of  the  four  orders 
indicated  in  Table  2,  permitting  an  examination  of  learning  or  inhibition.  That  is,  it  was 
anticipated  that  the  habits  acquired  in  coping  with  one  experimental  condition  might 
carry  over  in  to  another,  particularly  during  the  first  (non-crisis)  trial  block  of  the  new 
condition  (Gersick  and  Hackman,  1990).  In  the  first  experimental  session  teams 
encountered  their  first  crisis  during  the  second  trial  block.  It  was  anticipated  that  teams 
facing  high  severity  crises  would  notice  them  and  “shift  cognitive  gears”  before  teams 
facing  low  severity  crises.  As  Gersick  and  Hackman  (1990)  noted,  groups  faced  with 
severe  challenges  are  more  likely  to  notice  the  change  and  abandon  existing  habits 
than  are  those  faced  with  lessor  challenges.  They  also  argued  that  as  challenges 
become  more  frequent,  particularly  when  the  challenges  are  very  serious,  members 
become  less  and  less  likely  to  rely  on  habitual  patterns  of  behavior. 

Teams  were  offered  substantial  incentives  to  make  good  personal  and  collective 
decisions  as  a  means  of  achieving  good  performance.  The  criterion  for  individual 
performance  on  each  trial  was  the  accuracy  of  the  subject’s  estimate  of  their  own  (e.g., 
A  in  Figure  1)  unique  parameter’s  current  value.  The  criterion  for  collective  judgment 
was  the  team’s  accuracy  in  estimating  E,  the  difference  between  T  and  E. 

In  each  experimental  session  each  member  of  the  team  with  the  best  performance 
received  $20  for  that  session.  Members  of  the  second  best  team  received  $10,  and 
those  in  the  third  best  team  received  $5.®®  Members  of  all  other  teams  received 
nothing  for  that  session.  To  encourage  high  effort  throughout  the  experimental 
sessions  all  participants  were  also  eligible  for  overall  payments.  The  best  three  teams 
in  the  experimental  sessions  received  $50  for  each  member,  each  member  of  the  next 
best  three  teams  received  $20,  and  $20  was  paid  to  members  of  the  next  three  best 
teams.  Thus,  compensation  for  participation  could  range  from  nothing  to  $130. 

Data  about  the  individual  characteristics  of  each  team  member  were  collected  during 
the  training  session  for  two  reasons.  The  first  reason  was  to  achieve  greater  statistical 
control  through  procedures  such  as  the  use  of  covariates.  The  second  was  to  attempt 
to  clarify  mixed  results  from  earlier  studies.  It  is  unclear  whether  the  composition  of  the 
team,  for  example,  as  defined  in  terms  of  personality  profiles,  will  affect  decision 
processes  and  performance.  After  reviewing  experimental  studies  of  group 
performance.  Hare  (1994)  argued  that  the  mix  of  personalities  in  a  team,  together  with 
their  skills,  experience,  and  morale  (among  other  factors)  will  affect  problem  solving 
and,  more  generally,  group  productivity.  One  study  found  that  groups  designed  to  be 
heterogeneous  in  terms  of  the  need  for  dominance  were  more  effective  in  organizing 
communications  and  performing  their  task  efficiently  (Lampkin,  1972).  Other  research 
found  that  groups  with  heterogeneous  personality  profiles  solved  problems  better  than 
homogenous  groups  (Hoffman  and  Maier,  1961),  and  members  were  more  likely  to 


To  further  enhance  participants’  motivation  to  perform  well,  two  parallel  tournaments  were  actually 
conducted,  one  each  at  the  University  of  Iowa  and  Notre  Dame.  Thus,  for  each  experimental  session  a 
total  of  six  (of  the  13)  teams’  members  received  some  compensation. 
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challenge  one  another’s  incorrect  behavior  (Goldman,  et  al.,  1968).  (If  true  in  the 
complex  task  setting  of  this  experiment,  homogeneous  groups  would  be  outperformed 
by  heterogeneous  groups.) 

However,  there  is  some  evidence  that  homogenous  groups  communicate  more 
effectively,  cooperate  better,  and  experience  less  internal  conflict  (Bass,  1982;  Lodahl 
and  Porter,  1961;  Bass,  1965;  Hoffman,  1959). ).  Meta-analysis  reveals  that 
cooperative  orientations  produce  better  group  performance,  particularly  where  tasks 
require  exchanges  of  information  (Johnson,  et  al.,  1981).  If  a  crisis  should  cause  some 
to  become  more  competitively  oriented,  attempting  to  dominate  collective  processes 
and  outcomes,  the  resulting  conflicts  could  hinder  performance.  On  the  other  hand,  if 
members  have  and  retain  cooperative  orientations,  constructive  controversy  among 
members  should  help  achieve  high  performance  (Tjosvold,  1995,  page  90  ff.).^® 

Overall,  it  remains  unclear  whether  or  not  personality  matches  among  members  will 
make  a  difference  (Driskell  et  al.,  1987).  “[T]he  dominant  theme  in  the  empirical 
literature  is  ...  much  speculation  but  little  convergence.”  (Driskell,  et  al.,  1989). 

Research  in  other  contexts,  where  group  or  team  composition  is  not  an  issue,  may  be 
relevant  to  the  effects  of  personality  on  team  decision  making.  Recent  evidence  has 
shown  that  five  dimensions  (the  “big  five”  -  extroversion,  agreeableness, 
conscientiousness,  emotional  stability,  and  openness  to  experience)  reliably  capture 
individual  differences  (Digman,  1991).  Furthermore,  the  big  five,  particularly 
conscientiousness,  are  related  to  individual  task  performance  in  work  settings  (Barrick 
and  Mount,  1991).  A  few  individual  studies,  summarized  in  Driskell,  et  al,  (1989) 
suggest  that  these  dimensions,  particularly  stability  and  extroversion,  may  also  predict 
team  performance.  However,  in  complex,  multi-task  settings  like  ALCO,  not  all 
personality  traits  will  be  equally  relevant  to  all  decision  making  activities  and 
performance  (Driskell,  et  al,  1989).  Thus,  we  were  encouraged  to  use  the  Personality 
Characteristics  Inventory  (PCI),  an  instrument  which  has  demonstrated  substantial 
reliability  and  validity  (Barrick  and  Mount,  1993),  to  explore  how  the  mix  of  personality 
characteristics  in  each  team  might  relate  to  the  teams’  patterns  of  activities  as  well  as 
its  performance. 

Due  to  the  cognitive  aspects  of  the  task  situation  it  was  expected  that  individual 
cognitive  characteristics  could  also  be  important  compositional  considerations. 
Therefore  the  short  form  of  the  NFC,  Need  for  Cognition  questionnaire  (Cacioppo  et  al., 
1984)  was  also  answered  by  each  participant  at  the  end  of  the  training  session.  The 
short  form  has  demonstrated  reliability  comparable  to  the  original  (long)  form  (Cacioppo 
and  Petty,  1984)  and  has  exhibited  substantial  validity  (e.g.,  Tolentino,  et  al.,  1990; 
Sadowski,  1993)  and  provides  measures  which  are  unaffected  by  gender  (at  least  for 


Constructive  controversy  requires  members  to  express  aii  their  thoughts  and  emotions,  seek  to 
understand  others’  thoughts  and  emotions,  perceive  that  everyone  wili  share  the  same  fate,  attempt  to 
influence  others  and  be  subject  to  being  influenced,  exhibit  respect  for  other  members,  and  seek  soiutions 
that  are  mutually  satisfying. 
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college  students;  Sadowski,  1993).  NFC  assesses  an  individual’s  motivation  to  engage 
in  thinking  and  cognitive  endeavors.  The  NFC  is  also  useful  because  persuasion  is  a 
key  part  of  collective  decision  processes,  and  there  is  clear  evidence  that  the  need  for 
cognition  affects  persuasion  processes  and  their  outcomes  (Cacioppo  et  al.,  1983).  In 
terms  of  specific  aspects  of  the  decision  process,  it  was  reported  that  time  pressure 
affected  low  NFC  participants’  search  patterns,  but  not  those  of  high  NFC  participants’ 
search  (Verplanken,  1993).  Undertime  pressure  participants  used  search  strategies 
that  were  more  variable  in  amount  of  information  assessed  across  alternatives, 
indicating  the  use  of  more  heuristic  strategies,  than  when  not  under  time  pressure. 

Data  suggest  that  where  team  members  have  low  needs  for  cognition  they  will  probably 
experience  more  anxiety  about  communicating  with  others  (Wycoff,  1992).  There  is 
some  evidence  that  NFC  is  related  to  scores  on  the  MBTI  (Claxton  and  McIntyre,  1994). 
The  NFC  has  the  added  advantage  of  correlating  with  measures  of  general  mental 
ability  (Cacioppo  and  Petty,  1982)  like  the  ACT  and  grade  point  average  (Waters  and 
Zakrajsek,  1990).  Mental  ability  is  the  best  single  predictor  of  task  performance 
(Schmidt  at  al.,  1992).  For  all  these  reasons,  it  seemed  likely  that  the  higher  the  teams’ 
average  NFC  score,  the  better  it  would  be  likely  to  perform. 

To  explore  the  effects  of  other  composition  variables  a  demographic  questionnaire  was 
also  administered  to  all  participants  at  the  end  of  training.  It  collected  work-related  data 
on  years  of  part-time  work,  full  time  work,  supervisory  and  managerial  experience,  and 
self-assessments  of  important  job  skills  (for  example,  planning  others’  work,  decision 
making,  designing  work  systems,  working  as  a  team  member,  e-mail).  It  also  obtained 
data  on  participants’  ethnic  background,  native  language,  age,  sex,  and  undergraduate 
major.  Finally,  it  obtained  proxy  measures  for  intelligence:  grade  point  average,  and 
scores  on  college  admission  examinations.  Intelligence  has  been  moderately 
correlated  with  team  performance  in  pervious  research  (Driskell,  et  al.,  1989). 

Results 

Distributions  of  personal  data,  such  as  need  for  cognition  scores,  did  not  vary  by 
experimental  condition.®®  Due  to  the  small  number  of  observations  in  each 
experimental  condition  and  the  contamination  of  effects  due  to  decision  rule  and  type  of 
crisis  by  the  four  experimental  orders,  all  of  the  other  results  obtained  in  this  experiment 
must  be  treated  with  caution.  Furthermore,  all  confidence  intervals®^  for  effect  sizes 
contained  zero,  indicating  the  need  to  collect  more  data  before  drawing  firm 
conclusions.  The  following  graphic  displays  capture  the  most  noteworthy  performance 
and  communication  differences  which  were  obtained.  The  commentary  preceding  each 
figure  indicates  very  tentative  conclusions. 


In  retrospect,  this  is  not  surprising.  College  students  are  a  reasonably  homogeneous  population. 
Significance  tests  were  avoided  for  reasons  discussed  in  Schmidt,  1996. 
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Crisis  manipulations  were  effective.  Although  teams  in  all  conditions  performed  equally 
well  prior  to  crises,  crises  had  the  expected  effects  of  reducing  team  performance. 

Figure  2 

Mean  Performance  in  Sessions  1  and  2 
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Performance  tended  to  be  better  when  crises  were  less  severe,  despite  the  fact  that 
they  were  more  difficult  to  notice.  Decision  systems  had  less  impact.  For  example: 

Figure  4 


Mean  Performance  in  Sessions  1  and  2 
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The  patterns  in  the  following  three  figures  are  provocative.  The  first  indicates  that 
performance  for  all  teams  was  typically  comparable  prior  to  crisis. 

Figure  5 
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And  performance  returned  to  pre-crisis  levels  after  global  crises,  but  exceeded  pre¬ 
crisis  levels  following  local  crises  This  suggests  only  global  crises  spill  over  into  later 
non-crisis  activities. 


Figure  6 
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Differences  between  pre-  and  post-crisis  performance  depended  on  both  type  of  crisis 
and  decision  rule.  Dictatorship  was  most  useful  when  crises  are  local,  and  was  more 
dysfunctional  when  crises  are  more  extensive. 

Figure  7 
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Effects  of  decision  rule  and  type  of  crisis  appear  to  depend  on  previous  experience. 
For  example,  compare  the  results  for  the  first  two  experimental  sessions.  In  session  1 
teams  had  no  previous  experience  with  crises  of  any  kind  and  they  fared  best  under 
dictatorship  when  facing  the  most  severe  crisis. 

Figure  8 

Mean  Performance  in  Session  1 
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After  having  gained  some  experience,  team  performance  depended  on  an  interaction 
between  decision  system  and  severity  of  crisis.  Compared  to  the  consensual  system, 
dictatorship  was  less  useful  in  severe  crises  than  in  local  crises. 

Figure  9 

Mean  Performance  in  Session  2 
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During  crises  dictatorships  outperformed  consensus  teams  and  severe  crises  caused 
lower  performance  than  local  crises. 

Figure  10 

Mean  Crisis  Performance  in  Sessions  1  and  2 
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When  crises  affected  all  members,  more  messages  flowed  when  all  members  were 
compelled  to  agree  on  the  team’s  response  than  in  other  conditions. 


Figure  1 1 
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As  anticipated,  teams  required  to  reach  consensus  communicated  more  extensively, 
particularly  when  facing  severe  crises. 


Figure  12 


By  the  time  teams  had  completed  training  and  practice  sessions,  their  communications 
during  experimental  sessions  was  very  task  oriented  and,  as  seen  in  the  next  three 
graphs,  there  were  no  noticeable  differences  in  content  between  crisis  and  non-crises 
periods. 


Figure  13 


Mean  Messages  per  Pre-Crisis  Period 
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Figure  14 


Mean  Messages  per  Crisis  Period 
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Suggestions  for  altering  ALCO  processes  and  structures  were  more  common  during 
crises,  but  crisis  type  and  decision  rule  interacted  in  a  complex  fashion. 


Figure  16 


Mean  Strategic  Messages 
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In  most  conditions  the  number  of  messages  which  provided  other  team  members  basic 
information  was  quite  stable. 


Figure  17 
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Discussion  of  Experimental  Results 

As  noted  earlier,  these  data  require  replication  before  the  tentative  observations  which 
have  been  offered  can  be  accepted.  However,  the  following  points  seem  likely  to  be 
confirmed. 

Highly  motivated,  bright  people  find  it  difficult  to  focus  on  the  non-obvious  methods 
for  improving  performance.  There  were  several  indicators.  Strategic 
suggestions  were  far  less  common  than  informational  messages.  None  of  the 
members  sought  to  achieve  better  performance,  even  during  crises,  but 
attempting  to  review  their  peers’  past  activities.  Also,  the  density  of  flows  of 
messages  in  channels  was  essentially  the  same  whether  or  not  there  was  a 
crisis. 

Implicit,  but  central,  tasks  are  less  likely  to  affect  behavior  than  those  explicitly 
assigned.  For  example,  there  was  only  minimal  examination  of  past 
performance  reports  which  was  required  to  model  co-variation  between 
accuaracy  and  payback. 

Members  can  adapt  to  changes  in  each  person’s  relative  importance  to  team 
performance.  Subjects  did  learn  to  allocate  resources  in  proportion  to  impact  of 
each  member’s  individual  performance  on  team  accuracy. 

Failures  (e.g.,  in  communications  systems)  are  detected,  but  not  always  quickly. 

One  reason  is  that  teams  tended  not  to  institutionalize  lessons  from  past 
experience.  For  example,  teams  did  not  assign  responsibility  for  monitoring  key 
variables  to  specific  individuals  once  the  impact  of  those  variables  on 
performance  was  evident  from  experience. 

The  value  of  different  decision  systems  varies  depending  on  the  teams’  experience 
and  the  type  of  crisis  the  team  currently  faces.  However,  the  interaction  patterns 
do  not  conform  to  any  known  theory  and  are  not  readily  interpretable. 

Recall  the  earlier  argument  that  personal,  group,  and  organizational  success 
(sometimes  even  survival)  require  finding  an  appropriate  balance  between  efficiency 
and  readiness.  In  this  context,  the  experimental  results  suggest  that  participants  failed. 
For  the  most  part  they  focused  more  on  routine  tasks,  satisfying  themselves  with  only 
modest  adjustments  in  their  system  when  crises  arose.  Even  though  it  was  not  easy  to 
perform  even  the  routine  activities  with  great  efficiency,  adapting  organizational 
procedures  to  crises  was  even  more  problematic. 

It  is  most  likely  that  participants  in  this  experiment  found  it  difficult  to  recognize  and 
exploit  the  most  efficient  ways  to  modify  their  system  and  procedures  to  cope  with 
crises.  This  despite:  1)  being  selected  because  they  were  graduate  students  in 
business  averaging  more  than  two  years  of  full  time  (typically  non-managerial)  work 
experience,  2)  having  been  trained  in  the  system’s  properties,  and  3)  having  been 
warned  to  concentrate  as  much  on  system  management  and  innovation  as  on  more 
obvious  individual  and  team  task  demands.  In  fact,  once  they  made  discoveries  about 
the  kinds  of  crises  they  faced  they  appeared  to  be  content  to  make  only  simple 
adjustments.  Aggressive  efforts  to  discover  and  use  additional,  more  complex  coping 
tactics  or  strategies  were  uncommon.  These  behavioral  patterns  are  consistent  with 
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Gersick  and  Hackman’s  (1990)  speculations  that  people  are  highly  attracted  to  routines, 
finding  them  easy  to  adopt  and  difficult  to  abandon.  As  Hackman  and  Morris  (1975) 
also  observed,  teams  exhibited  few  signs  of  strategic  thinking:  planning  for  coping  with 
new  conditions  in  different  ways  that  were  used  in  routine  conditions  was  rare. 

Gersick  and  Hackman  speculated  that  teams  may  find  so  much  comfort  in  existing 
routines,  and  so  many  risks  or  costs  in  attempting  to  make  adjustments,  that  they 
unconsciously  increase  the  intensity  of  those  dominant  habits  when  aroused  by  a  crisis, 
consistent  with  other  manifestations  of  social  facilitation  (Zjonc,  1965)  in  individual 
behavior.  This  would  also  be  predicted  by  the  Icarus  paradox  exhibited  by  several 
leading  firms  (Miller,  1990). 

More  aggressive  response  efforts  had  been  expected  for  two  major  reasons.  First,  the 
crisis  conditions  had  been  pre-tested  and  usually  caused  team  performance  to 
plummet.  Thus,  members  could  hardly  fail  to  notice  the  situation  and  think  about  the 
need  to  find  new  ways  to  respond  (Weiss  and  llgen,  1985).  Second,  aggressive 
responses  were  expected  because  participants  were  highly  motivated,  not  only  by  the 
large  potential  earnings  available  in  the  tournament,  but  by  their  concern  for 
demonstrating  their  administrative  abilities  were  at  least  equal  to  their  peers’. 

These  results,  like  those  of  Hackman  and  Morris  (1975),  and  the  organizational  inertia 
reported  by  Miller  and  Friesen  (1984),  do  not  provide  many  causes  for  optimism.  It 
appears  that  imaginative  and  aggressive  exploration  of  new  or  additional  system 
designs  for  coping  with  crises  more  quickly  or  at  less  cost  will  not  be  observed  in  the 
absence  of  quite  large  incentives.  Can  it  be  possible  that  it  is  necessary  to  experience 
the  real  possibility  of  personal  bankruptcy,  death,  or  similar  extreme  threats  before 
initiating  intense  search  for  new  responses? 

Informal  discussions  of  these  experimental  results  with  experienced  managers  and 
senior  executives  reveal  that,  despite  (perhaps  due  to)  their  own  experience,  they  have 
the  same  tendencies.  When  they  do  think  in  terms  of  altering  their  systems  in  response 
to  threats,  their  actions  appear  to  demonstrate  reliance  on  the  availability  heuristic 
(Tversky  and  Kahneman,  1972).  That  is,  they  “benchmark”  (imitate)  what  they  consider 
to  be  the  best  practices  of  other  organizations.  For  example,  in  the  past  decade  firms 
typically  responded  to  large  increases  in  competitive  pressure  by  cost  reduction 
strategies  like  downsizing  rather  than  revenue  enhancing  strategies. 

If  administrative  experience  is  unlikely  to  induce  more  numerous  or  more  sophisticated 
approaches  to  forestalling  crises  before  they  occur  or  coping  with  crises  when  the  arise, 
graduate  education  in  management  may  not  be  a  satisfactory  alternative.  Participants 
in  the  experiment  were  drawn  from  two  of  the  leading  MBA  programs  in  the  United 
States.  However,  neither  program  emphasizes  crisis  management  or  the  case  method. 
Given  that  senior  administrators  confess  to  not  anticipating  crises  or  responding  in 
complex  fashion  to  crises,  one  must  wonder  how  much  could  be  accomplished  by 
participants  whose  graduate  training  had  focused  on  cases  involving  crises.  Certainly 
all  forms  of  training,  including  executive  short  courses  on  leadership,  reengineering 
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processes,  and  communications  and  negotiation  skills,  lack  the  visceral  impact  of  being 
a  member  of  a  team  or  organization  facing  extreme  threat. 

Use  of  ALCO  has  implications  for  those  who  would  conduct  similar  experiments.  Most 
researchers  are  likely  to  exhibit  overconfidence  with  respect  to  several  necessary 
activities.  An  appropriate  simulator  must  be  built  and  debugged.  Designs  must  be  very 
carefully  thought  out  and  efficient.  (Even  within-team  designs  like  the  one  reported 
here  only  achieve  a  threshold  of  efficiency.)  Longitudinal  effects  are  likely  in  real 
organizations,  but  are  difficult  to  include  in  experimental  simulators.  The  ease  with 
which  viable  combinations  of  simulator  parameters  can  be  established  for  an 
experiment  will  be  overestimated.  Mature  and  experienced  people  must  be  persuaded 
to  commit  substantial  time  as  subjects  for  relatively  modest  personal  returns. 
Researchers  must  find  enough  of  them  who  can  meet  at  one  time  to  achieve 
reasonably  robust  results  (something  not  achieved  in  Phase  3).  Researchers  will 
underestimate  the  time  it  will  take  subjects  to  learn  to  use  the  system  effectively. 
Funding  agencies  should  beware  of  proposals  which  are  overly  optimistic  in  these  and 
similar  respects. 

There  is  a  motivational  advantage  to  using  simulators  like  ALCO  for  experiments.  Most 
participants  become  very  involved.  This  is  beneficial  for  recruiting  subjects  and  eliciting 
their  peak  performance  over  long  experimental  sessions.  Unfortunately,  software 
development  must  constantly  accelerate.  For  example,  today’s  subjects  prefer  icon 
driven  systems.  (Unfortunately  from  this  perspective,  ALCO  is  a  DOS,  menu  driven 
system).  Adaptation  and  even  reprogramming,  of  existing  simulators  like  SIMNET 
(Alluisi,  1991;  Miller  and  Thorpe,  1995)  may  be  required. 

Conclusions  from  Phases  1-3 

Had  the  project  found  solutions  to  the  ICARUS  language’s  problems,  it  would  have 
significantly  enlarged  the  range  of  situations  to  which  discrete  event  simulation  can  be 
applied.  Incorporation  of  expert  systems  into  simulation  models  could  have  an  impact 
not  only  on  organizational  simulations  and  crisis  modeling,  but  on  conventional  logistical 
and  financial  simulations.  Simulation  efforts  like  Carley’s  (1986,  1991,  1992;  Carley 
and  Prietula,  1994)  shoud  be  encouraged  because,  even  though  they  don’t  have  the 
flexibility  ICARUS  would  provide,  they  do  include  variables  from  the  individual  through 
organizational  environment  levels.  However,  their  value  will  be  limited  as  long  as  they 
continue  to  not  be  validated  with  experimental  studies. 

It  has  been  nearly  10  years  since  Freeberg  and  Rock  (1987),  following  their  attempt  to 
derive  a  taxonomy  of  team  performance  based  on  meta-analysis  of  117  studies,  stated, 
“[Tjhere  have  been  glaring  deficiencies  in  the  team  performance  literature.”  Among 
those  they  emphasize  are  “methodological  quality  control”  (particularly  among  studies 
which  are  published  in  technical  reports  but  not  peer-reviewed  journals),  especially 
better  study  designs.  This  project  demonstrates  that  their  ideas  about  good  design 
should  be  extended.  Specifically,  designs  are  needed  placing  far  more  emphasis  on 
those  variables  affecting  decision  processes  which  are  inherent  in  the  fabric  of 
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organizational  activities.  Those  designs  should  include  factors  at  the  individual  level 
and  at  the  group,  organizational,  and  organizational  environment  levels  as  well. 

Without  extended  designs  incorporating  all  these  levels  it  is  impossible  to  determine 
whether  the  keys  to  improved  organizational  decisions  have  really  been  rigorously 
studied. 

More  than  better  designs  are  needed  however.  Support  for  the  conduct  of  experiments 
in  simulators  like  ALGO  needs  to  be  expanded  if  strong  induction  is  the  goal  of 
research.  Consider  a  design  which  uses  ALGO  to  create  an  organization  containing  a 
hierarchy  of  roles  involving  12  participants.  If  each  participant  must  be  trained  and  then 
work  in  only  one  two-hour  experimental  session,  and  if  each  participant  is  to  be  a 
person  experienced  in  both  routine  and  adaptive  administrative  challenges,  the  difficulty 
of  recruiting  an  adequate  number  (at  least  720)®^  of  subjects,  much  less  giving  them 
meaningful  incentives  to  excel,  threatens  to  forestall  the  entire  effort.  Only  in  contexts 
where  one  or  more  organizations  sense  the  importance  of  improving  organizational 
decision  processes,  and  are  therefor  willing  to  encourage  participation  in  an  extensive 
experimental  study,  are  meaningful  experiments  likely  to  be  conducted. 

Ironically,  influential  organizational  members  may  not  sense  how  pivotal  decision 
processes  are  in  balancing  demands  for  routine  efficiency  and  readiness.  Thus,  it 
seems  vital  to  accelerate  careful  analyses  of  organizational  decision  making  in  salient 
situations.  A  good  example  of  the  kind  of  research  which  might  motivate  increased 
support  for  ambitious  experiments  is  Feynman’s  work  (1992)  on  the  Rogers 
Gommission.  He  probed  relationships  between  individual,  group  and  organization 
behaviors  more  aggressively  than  most  of  the  other  members  of  the  Gommission. 
Another  example  is  Perrow’s  (1984)  moment  by  moment  analysis  of  the  Three  Mile 
Island  incident.®®  A  third  set  of  examples  can  be  found  in  Miller  (1990).  Two  common 
themes  which  become  apparent  in  these  kinds  of  sources  are:  1)  the  failure  of  those 
making  critical  decisions  to  think  systemically  rather  than  locally,  and  2)  apparent 
unconcern  for  reviewing  or  auditing  the  nature  and  quality  of  on-going  decision 
processes  in  real  time.®^ 

Regrettably,  the  validity  of  these,  and  other,  lessons  from  case  studies  has  not  been 
rigorously  established.  Not  only  are  there  no  experimental  data  to  support  them,  there 
are  also  no  comparative  case  studies.  While  there  are  well  known  dangers  to  “ex-post 
facto”  research  designs  (Gampbell  and  Stanley,  1963),  comparative  studies  would  be 
less  likely  to  permit  common  inductive  errors  (such  as  inferring  the  quality  of  a  decision 
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This  is  a  minimum  (and  rough)  estimate.  It  assumes  (only)  two  levels  of  (only)  two  independent 
variables  with  12  persons  per  observation  (each  simulated  organization  constituting  an  observation).  It 
also  assumes  that  robust  estimates  of  effect  sizes  will  require  approximately  15  observations  in  each 
experimental  condition. 

63 

Perrow’s  (1990)  analysis  (with  Guillen)  of  decisions  about  how  to  handle  the  AIDS  crisis  may  do  less  to 
increase  concern  with  organizational  decision  processes.  It  has  a  much  more  partisan  or  adversarial  tone 
than  his  earlier  (1982)  analyses. 

Even  retrospective  consideration  of  the  nature  of  organizational  decision  processes  is  very  unusual  in 
these  kinds  of  sources. 
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process  from  how  good  the  results  turned  out  to  be  -  Baron  and  Hershey,  1988).  And 
results  of  comparative  studies  would  provide  better  guidance  for  the  design  of  the 
“large”  experiments  which  are  needed  for  strong  induction. 

Given  the  uneveness  and  “thinness”  of  past  research  that  was  apparent  in  the  macro 
analysis  of  Phase  1 ,  it  is  difficult  to  make  specific  recommendations  for  “large” 
experiments.  It  does  seem  clear,  however,  that  they  need  to  incorporate  at  least  two  of 
the  following  levels  of  variables;  individual  differences  (e.g.,  expertise,  mental  ability 
and  personality),  social  (e.g.,  cohesion,  extra-role  behaviors  like  helping,  and  conflict 
management  procedures),  structural  (e.g.,  distribution  of  decision  responsibilities 
among  members,  communications  centrality,  and  status  hierarchies),  and 
environmental  (e.g,  stability,  and  munificence). 

Other  factors  also  discourage  “large”  designs.  Only  a  small  minority  of  social  scientists 
receive  scholarly  training  that  crosses  disciplinary  boundaries.  Consequently  they  may 
not  appreciate  research  outside  their  specialty  nor  know  how  to  collaborate  with 
scholars  from  other  disciplines.  Currently,  peer-reviewed  social  science  journals, 
particularly  those  emphasizing  experimental  research,  fail  to  encourage  submission  of 
studies  with  more  than  two  or  three  independent  variables.  Nor  are  most  social  science 
journals  receptive  to  cross-  or  multi-disciplinary  efforts.  Coupled  with  the  current  tenure 
incentive  system  which  tends  to  emphasize  collaboration  among  researchers  from  the 
same  discipline  on  projects  with  few  independent  variables,  designs  will  tend  to  be 
rigorous  but  too  “small”  to  provide  insights  about  group  or  organizational  decision 
making.®® 

If  adequate  experiments  are  mounted,  their  findings  will  need  replication.  While  there 
are  some  programmatic  efforts  (e.g.,  Davis,  Stasser)  in  today’s  “smaller”  experiments, 
efforts  which  permit  the  same  experimental  contrasts  to  be  made  in  more  than  one 
study,  they  are  not  common.  Unfortunately,  considering  the  resource  requirements  and 
other  barriers  to  “large”  experiments  on  organizational  decision  processes,  their 
replication  is  not  to  be  expected.  This  despite  the  fact  that  even  relatively  minor 
differences  between  apparently  identical  studies  can  produce  very  different  results 
(Vidmar  and  Hackman,  1971).  Multiple  studies  of  the  same  hypothesis  also  need  to  be 
encouraged  so  that  meta-analyses  can  be  applied  to  establish  the  effect  sizes  of 
different  combinations  of  independent  variables  (Hunter  and  Schmidt,  1990, 

1994,1991). 

If  experimental  research  remains  focused  on  “small”  designs,  this  project  documents 
the  need  to  redirect  those  efforts  to  relatively  neglected  topics.  Research  on  decision 
making  has  emphasized  individual  cognition  for  more  than  thirty  years.  Our  project 
indicates  that  even  if  every  individual  avoided  all  the  biases  and  errors  documented  by 
that  research,  team  and  organizational  decisions  are  unlikely  to  provide  adequate 


This  is  not  to  argue  that  aii  research  must  aiways  involve  complex  judgments  or  decisions.  It  is  true 
that  sometimes  “Meaningful  research  in  [organizational]  coordination  and  communication  requires  ... 
simple  tasks.”  (Carley  and  Prietula,  1994,  82). 
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guidance  for  improving  collective  decision  processes,  particularly  involving  dispersed 
teams  facing  crises. 

There  is  reason  to  hope  that  many  of  the  obstacles  encountered  in  this  project  will 
dissipate  over  the  next  few  years.  Teams  of  researchers  from  different  disciplines  can 
more  readily  collaborate  as  electronic  mail  and  more  advanced  collaborative  tools 
become  endemic.  If  the  medical  model,  where  multidisciplinary  teams  are  more 
common,  is  encouraged  in  the  social  sciences,  projects  like  this  will  be  more  likely. 
Economic  (e.g.,  high  salary  costs),  demographic  (e.g.,  a  smaller  student  base)  and 
technological  (e.g.,  world-wide  courses  offered  “by  wire”  over  the  internet  or  video 
satellite)  trends  paper  to  be  encouraging  change  in  universities.  And  universities  are 
increasingly  turning  to  corporations  for  support.  In  addition,  some  of  the  most 
interesting  and  useful  social  science  research  is  increasingly  conducted  in  corporations, 
think  tanks,  and  other  non-academic  institutions.  The  combination  of  these  events  may 
well  awaken  university  administrators  and  scholars  to  the  need  for  more  large  scale, 
multidisciplinary,  long  term,  and  rigorous  social  science  research  programs.  If  so,  the 
arguments  and  research  cited  in  this  report  may  be  useful  in  redirecting  some  of  those 
programs  toward  the  study  of  organizational  decision  systems. 
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Appendix  1 
Vincennes’  Crisis 

The  Navy’s  USS  Vincennes’  crisis  in  the  Iranian  Gulf  is  an  example  of  the  simultaneous  properties  of 
many  crises.  It  is  impossible  to  fully  empathize  with  those  who  experienced  it.  However,  by  reading  the 
following,  intentionally  unorganized,  run-on,  and  incomplete,  description  the  reader  may  be  able  to  sense 
how  different  a  real  crisis  can  be  from  experimental  settings. 

Decisions  were  made  by  many  individuals,  may  of  whom  were  unable  to  communicate  face  to  face.  Most 
participants  faced  multiple  decision  tasks,  each  being  a  distraction  from  the  others.  For  example,  the 
captain  needed  to  not  only  determine  whether  the  aircraft  on  radar  was  an  enemy,  but  also  how  to  combat 
the  surrounding  Iranian  gunboats.  The  situation  contained  unanticipated  and  unrecognized  linkages  that 
adversely  affected  the  decision  process.  For  example,  sailors  listening  to  Captain  Rogers  and  other 
officers  on  “walk-mans”  connected  to  the  communications  system  drained  electrical  power,  reducing 
volume  so  much  that  messages  could  not  be  heard  until  a  circuit  was  "flipped."  Also,  guns'  firing  caused 
electrical  pulses  that  produced  light  flickers  which  caused  an  officer  trying  to  read  civilian  flight  schedules 
to  miss  Iran  Air’s  flight  655  listing.  Mental  set  and  mental  maps  contained  beliefs  that  shaped 
interpretations  of  events.  For  example,  as  a  precaution,  planes  were  automatically  designated  "hostile"  at 
takeoff  from  Iran’s  Bandar  Abase  airfield  because  both  civilian  and  military  flights  originated  there.  In 
addition,  everyone  on  board  knew  that  Iran  viewed  them  as  serving  the  interests  of  “the  great  Satan.”  And 
the  scenario  was  reminiscent  of  the  USS  Stark’s  when  it  had  been  seriously  damaged  and  lives  had  been 
lost  by  an  attack  by  a  regional  power  just  over  a  year  earlier.  One  result  of  the  Stark  encounter  was 
standing  orders  to  fire  first  if  necessary  to  protect  crew  and  ship.  Personal  and  organizational  goals  were 
incompletely  aligned.  Rogers'  career  would  benefit  from  combat  experience,  but  U.S.  foreign  policy  would 
not  be  so  well  served.  Data  were  ambiguous.  For  example,  it  was  known  that  an  aircraft's  (civilian) 
identification  signal  could  be  readily  faked.  Radar  signals  were  difficult  to  interpret.  There  may  have  been 
overconfidence;  the  Vincennes  had  been  characterized  as  invincible,  a  “robocruiser.”  Communications 
were  simultaneously  excessive,  noisy,  and  inadequate.  Incoming  messages  from  other  ships  and  planes 
competed  with  those  sent  by  on-board  radar  observers.  Some  personnel  had  separate  messages  coming 
in  their  left  and  right  earphones.  There  was  no  exchange  of  messages  with  flight  655.  By  being  restricted 
to  electronic  media  many  subtle  messages  were  probably  unsent  or  distorted.  People  were  shouting  in 
the  command  and  control  center.  Incorrect  interpretations  were  reinforced  by  their  repetition,  whereas  an 
accurate  dissenting  view  was  “lost.”  No  one  could  identify  the  person  who  suggested  the  aircraft  was  an 
(Iranian)  F-14.  Misinterpretations  were  not  corrected,  so  the  captain  was  told  the  incoming  aircraft  was 
descending.  Time  pressure  was  extreme;  there  was  under  four  minutes  between  spotting  the  aircraft  and 
the  order  to  fire,  only  about  60  seconds  of  which  were  available  to  make  the  decision.  Conditions  were 
very  unpleasant.  Not  only  was  it  100  degrees  and  very  noisy,  but  the  ship  was  exposed  to  possible  mortal 
threat.  Despite  the  Navy’s  emphasis  on  keeping  cool  underfire,  emotions  were  elevated,  enhancing 
reliance  on  dominant  behaviors  and  beliefs. 
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Appendix  2 

Exannples  of  Simulation  Research  at  Different  Levels  of  Behavior 


Individual  Behavior: 

EPAM:  Serial  processing  recognition-based  expertise  including  learning  mechanism  for  acquiring  chunks 
stored  in  memory.  Predicts  behavior  reported  in  the  verbal  learning  literature.  Feigenbaum,  E.,  and 
Simon,  H.  (1984)  DPAM-like  models  of  recognition  and  learning.  Cognitive  Science,  Q,  305-336. 

ISACC:  Encodes  natural  language  statements  of  physics  problems  into  internal  images,  using  the  images 
to  produce  equations  which  it  solves.  Novak,  G.  (1976)  Computer  understanding  of  physics 
problems  stated  in  natural  language.  Technical  Report  Number  NL-30.  Austin,  TX:  Department  of 
Computer  Science,  University  of  Texas. 

UNDERSTAND:  Encodes  natural  language  descriptions  of  puzzles  into  internal  representations  that  are 
suitable  problem  spaces  for  GPS.  See  Hayes,  J.,  and  Simon,  H.  (1974)  Understanding  written 
problem  instructions.  In  Gregg,  L.  (Ed.)  Knowledge  and  Cognition.  Hillsdale,  NJ:  Erlbaum,  167-200. 

Group  Behavior: 

DISCUSS:  Models  effects  of  various  distributions  of  private  knowledge  in  decision  making  groups. 
Stasser,  G.  (1988)  Computer  simulation  as  a  research  tool:  The  DISCUSS  model  of  group  decision 
making.  Journai  of  Experimental  Social  Psychology,  24,  393-422. 

SIS:  Explores  relationships  between  decision  schemes  and  distributions  of  opinion  in  decision  making 
groups.  Stasser,  G,  Kerr,  N.,  and  Davis,  J.  (1989)  Influence  processes  and  consensus  models  in 
decision-making  groups.  In  Paulus,  P.  (1989)  Psychology  of  Group  Influence  (2nd  edition). 

Hillsdale,  NJ:  Erlbaum,  279-326. 

Organizational  Behavior: 

CONSTRUCT:  Investigates  changes  in  social/organizational  structure  which  arise  as  members  gather, 
interpret  and  communicate  data.  Individual  component  is  capable  of  learning.  Carley,  K.  (1988) 
Social  stability  and  constructionism.  Pittsburgh:  Social  and  Decision  Sciences  Working  Paper  Series, 
Carnegie  Mellon  University. 

CORBIN:  Battlefield  simulator  in  which  individual  behavior  is  often  in  the  form  of  small  expert  systems 
C3MUG,  Communications,  Command  and  Control  Military  Users  Group. 

GARCORG:  Interactive  model  explores  relationship  between  structure,  "event  theater"  and  decision 
making  capability  of  the  organization.  Carley,  K.  (1986)  Measuring  efficiency  in  a  garbage  can 
hierarchy.  Chapter  8  in  March,  J.,  and  Weissinger-Baylon,  R.  (Eds.),  Ambiguity  and  Command. 
Boston:  Pitman,  165-194. 

X-NET:  Reveals  which  organization  links  strengthen  and  which  links  atrophy  as  a  result  of  resource 
exchange  decisions  in  whatever  network  a  researcher  specifies.  Markovsky,  B.  (1987)  Toward 
multilevel  sociological  theories:  simulations  of  actor  and  network  effects.  Sociological  Theory,  5,  101- 
117. 
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Appendix  3 

Key  Words  or  Phrases  Used  in  Macro-Analysis 


ambiguity 

architecture,  organizational 

architecture,  cognitive 

assessment 

chance 

chaos 

choice 

cognitive  map 

cognitive  complexity 

cognitive  style 

cohesiveness 

confidence 

conflict,  cognitive 

conflict,  goals 

control 

coordination 

crisis 

decision 

diffuse  decision  or  problem 
dilemma 
distraction 
distributed  problem 
distributed  knowledge 
distributed  intelligence 
distributed  decision 
dynamic 

environmental  predictability 
error 

experience 

estimate  /  estimation 

experiment 

expert  system 

expertise 

feedback 

forecasting 

group  decision  rule  -  social  decision  scheme 

group  performance 

groupthink 

hostility 

hypothesis  generation 
image  theory 
information,  location 
information,  reliability 
information,  validity 
information,  quantity/load 
information,  completeness 
information,  timeliness 
interpersonal  conflict 
judgment 
linking-pin 
learning 
load,  work 


mental  model 

minority  influence 

network 

overconfidence 

overload 

probability 

redundancy  in  goals  (over  people  or  units) 
redundancy  in  mental  models  (over  people) 
redundancy  in  cues  (within  an  individual) 
redundancy  in  organizational  structures 
(e.g.,  status  congruency) 
redundancy  in  access  to  information  (over 
people) 

redundancy  in  communications  links 
(between  people  or  units) 
risk 

schema 

script 

search 

simulation 

social  decision  scheme 

stress  /  tension 

structure(s)  (of  organization) 

structure(s)  (of  access  to  information) 

surprise 

system 

thought  experiment 
threat 

time  pressure  /  deadlines 
turbulence 

turnover  (in  people,  units,  hardware) 
uncertainty 
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Appendix  4 

Criteria  for  Deletion  of  Studies  in  Macro-Analysis 

If  any  of  the  following  characterized  a  study  it  was  eliminated  from  consideration. 

abnormal  psychology  studies, 
developmental  or  age  studies, 
studies  not  reported  in  English, 
studies  involving  lexical  tasks, 

research  on  trivial  signal  detection,  visual  or  auditory  reaction  time  tasks, 
gender  studies, 

investigations  of  the  effectiveness  of  college  career  choice  aids, 
experiments  using  simple  recall  tasks, 

research  reporting  the  development  of  measures,  assessments  of  measures'  reliability  or  validity,  or 
other  investigations  of  a  measure's  psychometric  properties, 
investigations  using  trivial  ergonomics  (e.g.,  1  button  vs.  multi-button  vending  machines)  or  motor 
skills, 

clinical,  clinical  technique  development  studies, 

strictly  physiological  studies, 

experiments  with  simple  recognition  tasks, 

sexual  function  or  dysfunction  research, 

studies  of  highly  specific  tasks  (e.g.,  highway  curve  driving), 

studies  previously  considered  in  editing  other  searches, 

alcohol/drug  abuse  studies, 

quality  of  instruction  studies, 

studies  not  published  in  top  tier,  peer-review  journals, 

nonexperimental  studies,  and  non-empirical  articles,  such  as  literature  reviews  and  essays. 
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Appendix  5 

Journals  Sampled  to  Verify  PsycLIT  Search  Results 


Management: 

Industrial  Crisis  Quarterly 

Organizational  Behavior  and  Human  Decision  Processes 
Academy  of  Management  Journal 
Administrative  Science  Quarterly 
Journal  of  Consumer  Research 
Organization  Science 

Psychology: 

Journal  of  Applied  Psychology 
Acta  Psychologica 

Journal  of  Personality  and  Social  Psychology 
Journal  of  Experimental  Psychology 
Journal  of  Experimental  Social  Psychology 
Cognition 

Psychological  Review 

Psychological  Bulletin 

Annual  Review  of  Psychology 

Advances  in  Experimental  Social  Psychology 

Engineering: 

IEEE  Transactions  on  Systems,  Man  and  Cybernetics 
International  Journal  of  Man-Machine  Studies 

Sociology: 

Annual  Review  of  Sociology 

Social  Psychology  Quarterly  (Sociometry) 

Social  Networks 

Journal  of  Mathematical  Sociology 

Communications: 

Human  Communication  Research 
Small  Group  Research  (Small  Group  Behavior) 
Communication  Monographs 
Communication  Studies 

Qther: 

Journal  of  Conflict  Resolution 
Behavioral  Science 

Journal  of  Economic  Behavior  &  Organization 
Human  Computer  Interaction 
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Appendix  6 

Consolidated  Key  Word  Sets  -  Second  Matrix 


Set  1 :  Process  Factors 

assessment  /  evaluation  judgment 

choice  /  decision  knowledge  /  information 

communicate  /  coordinat(e)  /  control  search  /  problem  finding  /  hypothesis 

estimate  /  estimation  generation 

forecast(ing)  /  predict(ing)  memory  /  learning 


Set  2:  Context  or  Outcome  Factors 
achievement  /  accuracy 
error 

calibration  /  resolution 

ambiguity 

risk  /  uncertainty 

chance 

channel 

chaos  /  dynamic  /  turbulence 
cognitive  complexity  /  cognitive  style 
cognitive  map  /  mental  model  /  policy 
confidence  /  overconfidence  / 
underconfidence 
conflict  /  dissent 
crisis  /  emergency 
deadline  /  pressure  /  load 
diffuse  /  distributed  /  decentralized 
dilemma  /  public  good  /  free  rider 
minority  influence 
distraction 

experience  /  expertise 
cohes(ion)  /  morale  /  satisfaction 
expert  system 
feedback 

groupthink  /  conformity 
heuristic  /  decision  rule 


hostility  /  environmental  predictability 

image  theory 

influence  /  persuasion 

linking  pin 

locus 

location 

network 

structure 

architecture 

organization 

system 

overload 

performance  /  efficiency 
quality  /  speed  /  cost 
power  /  status 
probability  /  frequency 
schema  /  script 

shared  /  redundancy  /  duplic(ation)  /  overlap 
simulation  /  experiment 
social  /  interpersonal 

social  decision  scheme  /  group  decision  rule 

stress 

tension 

surprise  /  shock  /  threat 
turnover 


************ 


Groupings  of  Set  2  Variables  (Rows  of  third  matrix): 

"Accuracy"  -  achievement  or  accuracy,  error,  and  calibration  or  resolution. 

"Risk"  -  ambiguity,  risk  or  uncertainty,  chance,  and  probability  or  frequency. 

"Environment"  -  chaos  or  dynamic  or  turbulence,  and  hostility  or  environmental  predictability. 

"Mental  Model"  -  cognitive  complexity  or  cognitive  style,  cognitive  map  or  mental  model  or  policy, 
experience  or  expertise,  image  theory,  and  schema  or  script. 

"Crisis"  -  crisis  or  emergency,  stress,  and  surprise  or  shock  or  threat. 

"Outcomes"  (other  than  "accuracy")  -  cohesion  or  morale  or  satisfaction  and  quality  or  speed  or  cost. 

"Collective"  -  Conflict  or  dissent,  diffuse  or  distributed  or  decentralized,  minority  influence,  groupthink 
or  conformity,  influence  or  persuasion,  network,  structure,  architecture,  organization,  system, 
power  or  status,  shared  or  redundancy  or  duplicate,  or  overlap,  and  SDS  (social  decision  scheme) 
or  group  decision  rule. 

“Miscellaneous”  -  The  remaining  17  items. 
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Appendix  7 

Key  Components  of  ICARUS: 

The  generic  event  handler.  This  component  controls  the  order  in  which  the  simulator  processes  events  as 
the  scenario  unfolds.  This  is  based  on  a  translation  of  a  well  tested  package  originally  written  in 
Pascal.  The  Ada  version  has  been  thoroughly  tested. 

The  random  number  generator.  This  is  used  to  introduce  elements  of  chance  into  the  simulation,  for 
example,  in  the  modeling  of  the  probability  that  when  a  person  reads  a  meter,  they  will  misread  it,  or 
the  probability  that  when  a  person  tells  something  to  another,  they  will  be  misunderstood.  This  is 
based  on  a  well  understood  standard  package,  and  the  Ada  version  has  been  tested  to  conform  to  the 
standard. 

The  generic  list  manager.  This  component  is  used  to  manage  groupings  such  as  the  lists  indicating  who 
is  where,  or  the  lists  of  rules  maintained  for  each  person.  This  has  been  thoroughly  tested. 

The  name  manager.  This  component  is  used  to  handle  the  printable  names  of  model  components  so  that 
the  output  from  the  simulation  is  in  terms  readily  understood  by  humans.  This  has  been  thoroughly 
tested. 

The  lexical  analysis  package.  This  is  used  to  aid  in  processing  textual  material  describing  the  model. 

This  is  based  on  well  tested  Pascal  code  and  has  been  thoroughly  tested. 

The  common  types  package.  This  contains  definitions  of  quantities  needed  throughout  the  simulator. 

The  population  manager.  This  contains  the  action  routines  and  data  structures  used  to  model  each 
participant  in  the  simulation. 

The  geography  manager.  This  package  contains  information  about  places  in  the  model.The  value 
manager.  This  package  supports  the  set  of  values  that  may  be  taken  on  by  the  variables  in  the 
model.  Currently,  only  numeric  and  discrete  symbolic  values  are  supported. 

The  variable  manager.  This  package  stores  the  variables  and  manages  their  interpretation. 

The  expert  system  package.  This  package  encapsulates  the  rule  base  used  by  each  simulated 
participant.  It  is  only  partially  written  and  incompletely  specified.®® 

The  knowledge  manager.  This  package  manages  the  system  of  beliefs  held  by  each  participant. 
Participants  may  hold  beliefs  about  any  variable  in  the  model 

The  expression  manager.  Each  rule  in  the  expert  system  involves  expressions  that  combine  beliefs  about 
certain  values  to  determine  what  actions  to  take.  This  package  handles  the  storage  and  evaluation  of 
such  expressions. 

The  action  manager.  Each  rule  in  the  expert  system  may  specify  the  actions  a  participant  will  take  when 
some  condition  is  met.  This  package  is  used  to  manage  such  actions,  but  it  has  not  yet  been  possible 
to  fully  specify  all  the  possible  reactions. 


®®  We  wished  that  standard  expert  systems  could  be  used,  but  our  model  requireed  that  each  participant 
be  able  to  have  different  beliefs  about  the  values  of  the  variables  in  the  model,  and  current  expert  systems 
are  ill-suited  to  this  context  and  difficult  to  use  as  components  of  larger  software  systems. 
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Appendix  8 
ALCO  Variables 


ALCO  INDEPENDENT  VARIABLES:®^ 

1.  Basic  Independent  Variables: 

a.  Instructional  manipulations  -  In  addition  to  all  the  independent  variables  which  follow,  an 
experimenter  may  use  custom  "help  screens"  to  manipulate  experimental  instructions  of  any  kind 
that  are  to  differ  between  sessions.  For  example,  if  running  three  teams  of  subjects  in  a  session, 
the  teams  may  be  told  they  are  competing,  cooperating,  or  instructed  to  do  their  best. 

b.  Number  of  trials 

c.  Number  of  team  members  (12  maximum) 

d.  Quality  of  information  per  amount  spent  -  "Variance  Factors"  or  VFACTOR  -  (manipulated  using  a 
formula);  may  be  the  same  or  differ  by  member.  Once  set  it  remains  fixed  throughout  the  session. 

e.  Average  size  of  variance  factors  -  "High  vs.  low"  linkage  of  amount  spent  on  "search"  and  quality  of 

information  obtained  from  it 

f.  Average  size  of  correct  value  for  each  variable  in  organizing  principle  -  normally  will  be  different  for 

each  member 

g.  Organizing  principle  -  Nature  of  the  mathematical  operators  and  member  weights  relating  each 

member's  estimates  to  the  most  accurate  team  estimate  -  Can  be  used  to  manipulate  relative 
importance  of  each  member's  accuracy  to  team  accuracy 

2.  Budget  Independent  Variables; 

a.  Pooled  versus  individual  budget(s) 

b.  Initial  budget  amount  for  each  member 

c.  Initial  budget  level  (average  if  each  member  has  a  budget)  -  Can  be  thought  of  as  initial  wealth  of 

team 

d.  Ability  to  transfer  funds  between  members  -  (same  for  all  members;  0  cost  for  each  transfer) 

e.  Minimum  cost/charge  required  to  acquire  information 

f.  Amount  of  additional  funds  to  be  injected  to  budgets  during  session 

g.  Trial(s)  in  which  injections  of  funds  (if  any)  are  to  take  place  (same  for  all  members) 

3.  Payoff  Independent  Variables: 

a.  Maximum  possible  payoff  to  team  for  perfect  accuracy  -  Can  be  thought  of  as  degree  of  monopoly 

power,  environmental  "support" 

b.  Reward/Payoff  factors  -  Amount  of  payoff  to  ALCO  as  a  function  of  team's  accuracy  of  estimation 

(manipulated  using  a  formula  involving  both  a  linear  and  a  quadratic  penalty  for  inaccuracy)  -  To 
what  extent  does  the  team's  payoff  vary  with  team  accuracy? 

c.  Payoffs  may  be  added  either  to  the  team's  pool,  or  distributed  to  individual  budgets  (if  the  latter  are 

part  of  the  design)  (P/I) 

d.  Proportion  of  the  total  team  payoff  received  by  each  player  (if  payoffs  are  distributed  to  individual 

budgets) 

e.  Distribution  dynamics  -  If  payoffs  are  distributed  to  members,  proportion  distributed  to  each 

member  can  change  over  trials  as  a  function  of  each  member's  past  accuracy  (relative  to  other 
members'  accuracies)  -  By  setting  "alpha  value"  to  0  there  are  no  dynamics;  setting  it  to  1  permits 
payoff  distributions  to  depend  solely  on  the  players'  accuracies  in  the  last  trial;  values  between  0 
and  1  use  an  exponential  smoothing  of  accuracies  for  several  trials  with  differing  emphasis  on  the 
last  distribution  and  accuracies. 

4.  Communications  Independent  Variables: 

a.  Possibility  of  communication  between  members  -  Can  any  one  way  channel  ever  exist  between 
any  pair  of  members? 


Notes  on  independent  variables: 

In  this  appendix  funds  and  budgets  refer  to  resources  available  to  members. 

Some  independent  variables  can  also  be  manipulated  between  trials. 

Certain  independent  variables  can  only  be  used  when  the  individual  budget  manipulation  is  used. 
Some  manipulations  can  only  be  used  if  communications  between  members  are  permitted. 
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b.  Initial  communications  network  (if  communication  is  possible)  -  specified  using  a  "from  -  to"  matrix 

to  create  (e.g.)  "wheel,"  "chain,"  "circle,"  etc. 

c.  Modifiability  (by  members)  of  initial  communications  network  (if  any) 

d.  Blocked  channels  -  Are  there  some  channels  that  can  never  be  opened? 

e.  Cost  per  "one-way"  channel  opening  (if  network  can  be  modified)  -  (paid  from  opener's  budget  if 
there  are  individual  budgets,  otherwise  paid  by  pool  budget)  -  Can  be  defined  separately  for  each 
opener-openee  pair 

f.  Cost  per  communication  sent  -  "rent"  (paid  from  sender's  budget  if  there  are  individual  budgets, 

otherwise  paid  by  pool  budget)  -  Can  be  defined  separately  for  each  sender-receiver  pair 

5.  Feedback  Independent  Variables  -  Select  which  items  are  to  be  used  for  each  trial  (same  items  for 
every  member): 

a.  Member's  estimate 

b.  Correct  estimate  for  member's  parameter. 

c.  Absolute  error  of  member's  estimate 

d.  Relative  error  of  member's  estimate 

e.  Estimate  of  team 

f.  Correct  team  estimate 

g.  Absolute  error  of  team 

h.  Relative  error  of  team 

i.  Payoff  of  team 

j.  Proportion  of  team's  payoff  received  by  member 

k.  New  (current)  budget 

6.  Auditing  Independent  Variables: 

a.  Possibility  of  auditing  others'  behaviors  (ability  can  be  defined  separately  for  each  pair  of  members) 
-  Only  information  about  the  most  recent  trial  can  be  obtained  from  an  audit 

b.  If  auditing  is  possible,  what  information  about  auditee  can  be  audited?  Any  combination  of  the 
following  is  possible: 

i.  Amount  spent  for  information  by  auditee 

ii.  Correct  answer  for  auditee's  parameter 

iii.  Estimate  of  auditee's  parameter  by  auditee 

iv.  Absolute  Error  in  auditee's  estimate 
V.  Budget  of  auditee. 

c.  Cost  per  audit  (paid  by  auditor's  budget  if  there  are  individual  budgets,  otherwise  paid  by  pool 

budget)  -  Can  be  defined  separately  for  each  auditor-auditee  pair 

ALCO  DEPENDENT  VARIABLES  -  The  following  data  are  automatically  collected.  Additional  dependent 
measures  are  often  obtained  by  using  paper  and  pencil  questionnaires. 

1.  Financial  Dependent  Variables: 

a.  Amount  spent  for  information  -  by  member  and  trial 

b.  Amount  spent  on  channel  creation  by  opener-openee  pair 

c.  Amount  spent  on  messages  by  sender-sendee  pair 

d.  Amount  spent  on  audits  by  auditor-auditee  pair 

e.  Amount  of  funds  transferred  by  pair  of  members  and  trial 

f.  Payoff  received  by  team  on  each  trial 

g.  Payoff  received  by  each  member  on  each  trial 

h.  Budget  level  (individual  or  pooled)  at  all  times 

2.  Communications  Dependent  Variables: 

a.  Number  of  one  way  channels  opened  by  openee-opener  pair 

b.  Number  of  messages  sent  by  sender-receiver  pair 

c.  Number  of  audits  by  auditor-auditee  pair 

d.  Contents  of  messages  -  Text  file  saves  all  transcripts 

3.  Performance  Dependent  Variables: 

a.  Estimate  of  own  parameter  value  for  each  member 

b.  Correct  parameter  value  for  each  member 
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c.  Absolute  error  of  member's  estimate  on  each  trial 

d.  Relative  error  of  member's  estimate  on  each  trial 

e.  Absolute  error  of  team's  estimate  on  each  trial 

f.  Relative  error  of  team's  estimate  on  each  trial 

g.  Time  duration  and  latency  measures  -  All  activities  are  time 

h.  Number  of  trials  (if  session  ends  due  to  depletion  of  budget) 


stamped 
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Addendum  1 
Project  Products 


Published  Reports: 

Jones,  D.  Simulation  of  information  flow  in  organizations.  Proceedings  of  the  IEEE  Winter  Simulation 
Conference,  Los  Angeles,  December  1992. 

Invited  Presentations: 

Rose,  G.,  and  Conlon,  E.  ALCO  -  An  Organizational  "Flight  Simulator."  Presented  to  the  TIMS/ORSA 
Mathematical  Organization  Theory  Workshop,  Chicago,  IL,  May  15-16,  1993. 

Rose,  G.  Learning  from  Experiments  on  Organizations:  Lessons  from  ALCO.  Presented  at  the 
national  meeting  of  INFORMS  (TIMS/ORSA),  April  23,  1995. 

Learning  from  Experiments  on  Organizations:  Lessons  from  ALCO.  Presented  at  the  national 
meeting  of  INFORMS  (TIMS/ORSA),  April  23,  1995. 

In  response  to  an  inquiry  by  Howard  Berg,  deputy  director  of  ASSET,®®  two  Ada  packages  developed  for 
this  project  were  contributed  to  ASSET.  One  is  a  pseudo-random  number  generator  package;  the  other  a 
generic  pending-event-set  management  package.  These  are  core  components  of  ICARUS  and  are 
applicable  to  many  other  simulation 


®®  ASSET  -  Asset  Source  for  Software  Engineering  Technology  was  organized  by  the  Defense  Advanced 
Research  Projects  Agency  (DARPA)  under  its  Software  Technology  for  Adaptable,  Reliable  Systems 
(STARS)  program.  DARPA  tasked  IBM  and  its  principal  subcontractor,  SAIC,  (Science  Applications 
International  Corporation)  to  establish  the  ASSET  reuse  library  to  serve  as  a  national  resource  for  the 
advancement  of  software  reuse  across  the  DoD.  ASSET'S  mission  is  to  provide  a  distributed  support 
system  as  a  focus  for  software  reuse  within  the  DoD  and  to  help  foster  a  software  reuse  industry  within 
the  United  States. 
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